Edge gateway HTTP-01 challenges stuck at ACME self-check (DNS/404/500/503) — NSO solver is live; control-plane vs member Challenge split

[ecv]

2026-07-13 rewrite. This OP originally diagnosed a stalled/frozen NSO solver. Live investigation from the prod hub + a reproduction of NSO's own federated client disproved that. NSO's solver is live and reconciling correctly; the block is elsewhere. The two earlier investigation comments are minimized as outdated. Verified current picture below.

Summary

Edge gateway custom-hostname certificates are not issuing. But the GatewayDownstreamCertificateSolver is not the cause — it is running, holds leadership, and reconciles the Challenges it can see, creating the expected HTTP01 solver HTTPRoute/HTTPRouteFilter. The Challenges are stuck at cert-manager's ACME HTTP-01 self-check, which is failing for heterogeneous reasons (DNS resolution failures, HTTP 404/500/503). Separately, the control-plane Challenge set NSO watches has no object newer than 2026-06-18, while edge member clusters carry fresh Challenges — a topology split that needs explaining.

Verified topology (reproduced NSO's client)

• The solver watches the Karmada control-plane apiserver (karmada-apiserver.karmada-system.svc.cluster.local:5443), reached via the pod's federated-kubeconfig + a projected SA token (network-services-operator-controller-manager, audience = that apiserver). Reproduced this exact client from the hub and queried it directly.
• That apiserver currently serves 16 gateway acme.cert-manager.io/v1 Challenges (issuerRef=ClusterIssuer/datum-gateway-http, type: HTTP-01), all created ≤ 2026-06-18 (two older, from March/May), all pending, and actively updated by cert-manager today (managedFields: cert-manager-challenges@2026-07-13T05:23Z).
• The solver's surviving http01-solver HTTPRoute/HTTPRouteFilter set on the edge maps 1:1 to these 16 control-plane Challenges.

What is actually happening

The solver is live, not stalled. Leader pod network-services-operator-controller-manager-849f57c79d-xwp54 (fresh, restartCount 0) logs the full happy path with 0 errors, and re-reconciles on cert-manager's live status updates:

Reconciling ACME challenge solver
Processing challenge for Gateway-owned certificate
Successfully retrieved key and token from cert-manager Challenge
HTTPRouteFilter reconciled
HTTPRoute reconciled

The Challenges are stuck at the HTTP-01 self-check, for mixed reasons (live status on the 16):

Failure reason (from status.reason)	Notes
dial tcp: lookup <domain> on 10.2.0.10:53: server misbehaving	cluster-DNS resolution failing for the self-check (e.g. app.datum.design)
wrong status code '404', expected '200'	e.g. javalords-ink…, swirls-k8s…
wrong status code '500'	e.g. pcep-lol…, tunnel-dmsrd…
wrong status code '503'	seen intermittently on the same objects
failed to perform self check	generic self-check failure

So even objects for which the solver has created a route still fail the self-check (404/500/503), and some never get that far (DNS). This is a data-path / self-check problem, not a "solver isn't creating routes" problem.

The real open question — control-plane vs member Challenge split

• The control-plane apiserver NSO watches has no Challenge newer than 2026-06-18. That, not a dead watch, is why no new solver routes appear after 06-18 — NSO faithfully serves what's there, and there's nothing newer.
• Meanwhile the edge member us-central-1-alice (a registered Karmada member) has 96 Challenges created 2026-07-13 02:43, which do not appear in the control-plane view. The 06-18 control-plane object javalords-ink-https-hostname-0-1-2137251722-2111718441 is even NotFound on that member.
• So there appear to be two Challenge populations (control-plane vs member) that don't reconcile. Where are new tenant gateway Challenges being created, and why don't they land in the apiserver NSO watches?

What to investigate next (owner)

1. Why the HTTP-01 self-check fails per-domain: cluster-DNS server misbehaving for control-plane cert-manager, and edge Envoy returning 404/500/503 on /.well-known/acme-challenge/<token> even where a solver route exists.
2. The control-plane ↔ member Challenge topology: is new tenant cert issuance supposed to create Challenges in the control-plane apiserver (where NSO + datum-gateway-http live), and if so why has nothing appeared since 06-18?
3. Whether the 06-18 ceiling correlates with the v0.22.0 (EG 1.8 / Gateway API 1.5, Modernize the gateway stack: Envoy Gateway 1.8 & Gateway API 1.5 #189/Envoy Gateway extension server #192, released 2026-06-18T01:07Z) change to the federated/aggregated wiring — but note this is about what gets into the apiserver, not the solver's watch, which is demonstrably live.

Refs

• datum-cloud/infra#3419 (field symptom tracker)
• AI Edge: eliminate slow/failed TLS cert provisioning at scale via Envoy Gateway extension server #216 (prior TLS-at-scale incident — different, already-fixed cause)
• feat: add automatic re-issuance of failed custom hostname certificates #259 / Unify or delineate cert-failure recovery: re-issuance (#259) vs health-gating (#212) #260 (re-issuance and its interaction — still a churn hazard against any stuck cert)

[ecv]

Live prosecution (prod hub, 2026-07-13 ~03:47Z)

Ran the confirm steps against prod. All three network-services-operator-controller-manager pods on v0.23.10 (started 2026-07-13 03:00–03:02, --cluster-sharding-enabled=true, lease prefix nso in kube-system).

Symptom confirmed live. Gateway path healthy; solver produces nothing despite active failing Challenges.

Check	Result
Gateway controller alive	✅ 90 / 105 / 113 gateway reconciles in the recent window across the 3 pods (plus a few connector)
Solver controller activity	❌ zero solver / challenge / acme / Reconciling ACME lines in any pod
Live cert-failure impact	✅ 16–17 "We couldn't issue a TLS certificate" messages per pod — real domains incl. swirls.ai, app.datum.design, agent.lkms.app
Singleton lease location	❌ no nso-prefixed Lease in hub kube-system — confirms the singleton leader lease lives on the Karmada aggregated apiserver, as described above

This isolates the fault to the singleton solver, not the gateway controller — matching the report.

What this run could NOT settle (hub visibility limits)

• Only the recent steady-state log window was readable; startup lines (leader election, Starting workers … downstream-certificate-solver, the initial Challenge LIST) had already scrolled off.
• The hub flux-MCP can't reach the Karmada aggregated apiserver, so confirm steps 1 (controller started + holds leadership) and 2 (Challenge watch established vs. dead-after-initial-LIST) remain open. Need the leader pod's startup logs or Karmada kubeconfig access.

Two possible causes remain, and this run can't tell them apart:

1. the solver controller never started on the leader, or
2. it started but its Challenge watch died after the initial LIST.

Next diagnostic is confirm step 1 — is the controller started and holding leadership. Cheapest check, and the one hub visibility can't answer.

[ecv]

Deeper investigation — the "does nothing / maybe not started" framing is wrong. It's a frozen Challenge informer serving phantom objects. (prod, 2026-07-13 ~05:14Z)

Got kubectl to the hub GKE cluster (gke_datum-cloud-prod…infrastructure-control-plane-prod), where the three v0.23.10 pods actually run, so their full logs are readable (the flux-MCP tail caps at ~750 lines ≈ 30s given these pods emit ~20k lines/hr).

Step 1 — is the singleton solver started + holding leadership? YES.

Leader is pod …-849f57c79d-xwp54 (fresh: created 2026-07-13T03:02, restartCount: 0). Its log shows the downstream-certificate-solver controller actively reconciling — full happy path, no errors:

Reconciling ACME challenge solver
Processing challenge for Gateway-owned certificate
Successfully retrieved key and token from cert-manager Challenge
HTTPRouteFilter reconciled
HTTPRoute reconciled

So the earlier "not started / not acting" hypothesis is wrong — it is started, holds leadership, and its reconcile loop runs. This also rules out the sub-hypothesis "never started on the leader."

Step 2 — is the Challenge watch alive? NO — the informer is frozen at the 2026-06-18 LIST snapshot.

Over a 47-minute window (04:27–05:14Z) the solver logged 59 Reconciling ACME challenge solver — every single one for the same object:
ns-4250da12…/javalords-ink-https-hostname-0-1-2137251722-2111718441. 59 reconciles, 59 HTTPRoute reconciled, 0 errors, 0 other challenges touched.

Then the killer cross-check against the edge (datum-us-central-1-alice):

Check	Result
The hot-looped challenge on the edge	NotFound — it was deleted (it's a phantom the solver still thinks exists)
Its HTTPRoute creationTimestamp	2026-06-18T03:06:27Z — never moves; the reconcile is idempotent no-op on a pre-existing route, which is why HTTPRoute reconciled logs but nothing changes
Live challenges in that same namespace	14 real ones, all created 2026-07-13T02:43, pending — the solver reconciled none of them
Newest http01-solver HTTPRoute edge-wide	still 2026-06-18T03:08:51Z, unchanged across two snapshots 1.5h apart

What this means

The singleton solver's Challenge informer is stuck on the ~2026-06-18 LIST. It keeps re-delivering cached objects from that snapshot — including at least one already deleted — so it busy-loops reconciling a phantom, while receiving no ADD events for the 95+ challenges created since and no DELETE for the ones removed. Routes therefore haven't advanced past 2026-06-18, and every new edge challenge 404s. This is the concrete form of "watch died after the initial LIST."

The sharper clue for the owner

The leader pod is fresh (03:02 roll today), yet its informer is serving 25-day-old data including deleted objects. A fresh pod's informer can't hold stale in-process state — so the frozen snapshot is coming from what the federated/aggregated apiserver serves to DownstreamCluster.GetCache()'s LIST/watch, not from NSO's in-memory cache. That also explains why plain restarts (incl. the v0.23.10 roll) don't recover it: a new informer re-attaches to the same frozen upstream stream. Points at the Karmada aggregation / federated-client path (consistent with the v0.22.0 EG 1.8 + Gateway API 1.5 modernization touching that wiring) rather than the solver controller code itself, which is behaving correctly given the objects it's handed.

Recommend: check the federated/aggregated apiserver's watch-cache for acme.cert-manager.io/v1 Challenge (resourceVersion it serves on LIST vs. current), and whether the aggregation layer is pinning an old RV / dropping the watch silently.