Stale hostname CNAME/ALIAS not reaped on hostname removal — breaks resolution (SERVFAIL)

[ecv]

Summary

When a custom hostname is removed from an HTTPProxy/Gateway (or its proxy binding is torn down), the DNS record the platform created for it (a CNAME for non-apex, ALIAS for apex, pointing at the proxy's *.datumproxy.net canonical name) can be left behind. The stale record then points at a canonical name that no longer exists — so the hostname fails to resolve (authoritative SERVFAIL) and can block a later resource from re-claiming it.

Warning

Confirmed via DNS, with live customer impact. Originally filed against the HTTPProxy-delete path — that path's cleanup is present and fixed since NSO v0.17.0 (prod v0.24.3). The live gap is on hostname/binding removal: the customer-hostname record survives after its proxy canonical is gone. Only formal step left is confirming the upstream proxy no longer lists the hostname (see Open question).

How the records flow (where reap must happen)

1. Upstream (project control plane) — gateway_dns_controller creates a per-hostname DNSRecordSet, Gateway-owned. On hostname removal it should be reaped by the label-based garbageCollectDNSRecordSets (diff desired-vs-existing, delete stale).
2. Downstream projection (host cluster, ns-<uid>) — a mirrored copy owned by the DNSZone + an anchor ConfigMap. GC'd only when its upstream is deleted (anchor cascade). Anchor-ownership here is by design, not the bug.
3. PowerDNS — dns-operator writes/removes the RRset and resolves ALIAS targets at query time; a live CR ⟹ the RRset is kept. If the ALIAS target is NXDOMAIN, PowerDNS returns SERVFAIL for the name.

Reaping a removed hostname therefore depends entirely on step 1 (upstream garbageCollectDNSRecordSets).

Evidence (prod — customer AB1889, ab.dk, project ab-generic-1qnl3i)

• Downstream DNSRecordSet ab-website-g2rfww-9f7feda5 = ALIAS www.ab.dk → loneliness-snake-9nkbf.datumproxy.net; Programmed=True, Accepted=True, created 2026-07-12, still present days later. Anchor present ⟹ upstream DNSRecordSet still exists.
• The proxy hostname was moved www.ab.dk → bare ab.dk (see infra#3407), yet the www.ab.dk record persists.

DNS confirmation (queried this session)

• www.ab.dk → SERVFAIL, authoritatively, on both ns1/ns2.datumdomains.net.
• Cause: the ALIAS target loneliness-snake-9nkbf.datumproxy.net → NXDOMAIN (confirmed on 8.8.4.4, 1.1.1.1, and authoritative) — the proxy canonical was torn down.
• Control: sibling media.ab.dk (ALIAS → a live proxy canonical) → NOERROR; datumproxy.net zone is healthy. So the zone/operator are fine — only www's target is dead.
• User impact: www.ab.dk is unresolvable for any resolver without a cached positive (ALIAS TTL 300s); stale caches mask it intermittently.

Net: the proxy binding/canonical for www.ab.dk was removed but its customer-hostname ALIAS was not reaped → the reap gap, now breaking resolution.

Open question (formal final confirmation)

Pull the upstream HTTPProxy ab-website-g2rfww spec.hostnames from PCP ab-generic-1qnl3i:

• www.ab.dk removed (expected, given the canonical is gone) → confirms garbageCollectDNSRecordSets isn't reaping on hostname removal → NSO bug.
• www.ab.dk still listed → the record is intentional but its target is dead — a different failure (canonical not maintained), still a bug.

Test-coverage gap (fold into the fix)

• garbageCollectDNSRecordSets is unit-tested in isolation (gateway_dns_controller_test.go:771,845), but no test drives the full create → remove-hostname reconcile round-trip.
• Gateway-delete finalizer cleanup (cleanupDNSRecordSets, gateway_controller.go:1676) and the HTTPProxy-delete DNS cascade are untested.
• No chainsaw e2e references DNSRecordSet at all.
• The fix should add: hostname-removal reap (upstream + projected downstream), Gateway-delete, and HTTPProxy-delete cases — asserting the record is gone and the hostname stops resolving.

Related

• https://github.com/datum-cloud/infra/issues/3407 — real-world ab.dk instance
• DNSRecordSet delete can orphan the PowerDNS RRset: no finalizer on PDNS controller + dead cleanup path dns-operator#58 — related latent PDNS restart-orphan (+ ALIAS-delete untested)
• https://github.com/datum-cloud/infra/issues/1770 — external-dns TXT orphan (same failure shape)
• Record-management origin: feat: automatic DNS record management for gateway hostnames #111, fix: store relative DNS label and correct record type for gateway-managed records #134

[ecv]

Closing this — I filed it too hastily, and independent verification refutes it.

I read the current main and checked the deployed version. The delete-side cleanup I claimed was missing has existed since v0.17.0 (Feb 2026) and is live in prod today (ghcr.io/datum-cloud/network-services-operator:v0.24.3). Records are cleaned up three ways, all same-cluster (the DNSRecordSet lives in the upstream project control plane alongside the Gateway):

1. Hostname removed from spec — garbageCollectDNSRecordSets runs at the end of every reconcile and deletes any managed DNSRecordSet not in the desired set (internal/controller/gateway_dns_controller.go).
2. Gateway deleted — finalizer gateway.networking.datumapis.com/gateway-controller triggers cleanupDNSRecordSets, which explicitly deletes the record sets before the finalizer is removed (internal/controller/gateway_controller.go).
3. HTTPProxy deleted — the proxy owns its child Gateway via a controller reference, so deleting the proxy garbage-collects the Gateway, which fires mechanism 2.

Introduced by cd4f009 ("fix: explicitly delete DNSRecordSets when Gateway is deleted") and c0606c9 ("go back to always deleting the dns record"), both first tagged v0.17.0.

On the ab.dk instance in https://github.com/datum-cloud/infra/issues/3407 (Issue 2): the ALIAS that "could not be deleted" was not this bug. That customer ran on v0.24.x, which has all three mechanisms. That failure is at a different layer — the actual record at the DNS provider (downstream replication / dns-operator), not the DNSRecordSet CR lifecycle in NSO. It's the same shape as https://github.com/datum-cloud/infra/issues/1770 (external-dns not cleaning orphaned records on delete) and stays tracked in that infra#3407 note, not here.

Not a bug. Sorry for the noise.

[ecv]

(closed prematurely)

[ecv]

Reopening — new evidence changes the picture (my close above, now hidden, was wrong).

The orphaned record in the wild (https://github.com/datum-cloud/infra/issues/3407, ALIAS www.ab.dk → loneliness-snake-9nkbf.datumproxy.net) is not the standard HTTPProxy→Gateway path I verified as fixed. The live DNSRecordSet (ab-website-g2rfww-9f7feda5) shows:

• Created 2026-07-12, still present days later; finalizer dns.networking.miloapis.com/finalize-dnsrecordset-downstream
• Owned by DNSZone ab-dk-pwibf0 + an anchor ConfigMap — not the Gateway
• A downstream projection (meta.datumapis.com/upstream-cluster-name: cluster-_ab-generic-1qnl3i, upstream-namespace: default), created via the AI-Edge path

The three cleanup mechanisms in gateway_dns_controller / gateway_controller key off Gateway ownership, so they never touch AI-Edge-projected record sets owned by DNSZone + anchor. That's the real gap this issue should track — re-scoping toward the AI-Edge / downstream-projected DNSRecordSet lifecycle.

Ruled out — dns-operator is correct here: it keeps the PowerDNS ALIAS because the DNSRecordSet CR is still alive (it deletes an RRset only when no live CR owns the tuple). The separate, latent restart-orphan risk at that layer is now tracked in datum-cloud/dns-operator#58.

Still pinning the exact sub-layer — upstream DNSRecordSet stale-and-not-reaped vs downstream projection finalizer stuck. Will update once I've pulled the upstream state from ab-generic-1qnl3i.

[ecv]

Working hypothesis

The stale record isn't reaped because the AI-Edge DNS path creates DNSRecordSets owned by the DNSZone + a projection "anchor", not by the Gateway — while the cleanup logic (garbageCollectDNSRecordSets) only deletes record sets it finds owned by the Gateway. So when a hostname is removed from a proxy on the AI-Edge path, the per-hostname record set is never in the Gateway-owned set the GC walks, and it survives.

Evidence so far (prod, ab.dk / project ab-generic-1qnl3i):

• Downstream DNSRecordSet ab-website-g2rfww-9f7feda5 (ALIAS www.ab.dk → loneliness-snake-9nkbf.datumproxy.net) is live, Programmed=True, Accepted=True, created 2026-07-12, still present.
• Owners: DNSZone ab-dk-pwibf0 (controller) + anchor ConfigMap anchor-95f1bd22-… — no Gateway owner.
• The anchor's presence implies the upstream DNSRecordSet in the PCP still exists.

Open confirmation (in progress): pull the upstream HTTPProxy ab-website-g2rfww spec.hostnames + the upstream record's ownerRefs from the PCP.

• www.ab.dk absent from the proxy but record persists → confirms the reap gap above.
• www.ab.dk still present → record is correctly maintained; the orphan is stale proxy config, and this issue narrows/closes.

[ecv]

Correction to the hypothesis above

A test-suite audit refined this — the anchor-ownership I flagged is by design at the downstream layer, not the bug.

• Gateway/HTTPProxy-hostname DNSRecordSets are created upstream (PCP), Gateway-owned (gateway_dns_controller.go:304 — owner-ref for GC). There is no separate anchor-owned creation path for gateway hostnames; the only anchor/DNSZone-owned records are iroh Connector records.
• The downstream copy I saw in prod (owned by DNSZone + anchor ConfigMap) is the projection. It's reaped when its upstream is deleted, via the anchor cascade (DeleteAnchorForObject), which fires only on whole-owner deletion.

So reaping a removed hostname depends entirely on the upstream, label-based garbageCollectDNSRecordSets deleting the Gateway-owned record when the hostname leaves the desired set. Refined fork for ab.dk:

• www.ab.dk still in the upstream proxy's desired hostnames → record correctly kept (not a bug; stale proxy config).
• www.ab.dk removed but the upstream record survives → garbageCollectDNSRecordSets isn't reaping on hostname removal — the actual NSO bug.

Test-coverage gap (fold into the fix)

• garbageCollectDNSRecordSets is unit-tested in isolation (gateway_dns_controller_test.go:771,845), but no test drives the full create-then-remove-hostname reconcile round-trip — a reconcile-level reap failure would go uncaught.
• Gateway-delete finalizer cleanup (cleanupDNSRecordSets, gateway_controller.go:1676) — untested.
• HTTPProxy-delete → child-Gateway DNS cascade — untested (the finalizer test covers EnvoyPatchPolicy/anchor only).
• No chainsaw e2e references DNSRecordSet at all.

A regression test for the fix should: create a proxy with a hostname, confirm the record, remove the hostname, and assert the record is reaped (upstream + projected downstream) — plus a Gateway-delete and HTTPProxy-delete case.

[ecv]

Plan to fix this, front-loading what we can do today.

First, confirm the root cause in a dev cluster: create a proxy with a hostname, remove the hostname, and check the record is reaped. The prod evidence points somewhere specific — the stale record was never even touched by cleanup (its state didn't change all session), so the likely cause isn't a broken reaper but that these records fall outside the set cleanup looks at. The dev repro tells us whether it's a plain controller bug or a selection/labeling gap that also strands already-created records.

Then the fix: make hostname-removal cleanup cover every managed per-hostname record, for both new and pre-existing ones, so removing a hostname always reaps its record — upstream, the projected copy, and PowerDNS.

Tests ship in the same PR — the missing coverage is what let this rot. A full create → remove-hostname round-trip, plus gateway-delete and proxy-delete cleanup, plus an end-to-end test (there is currently none touching DNS records at all).

Release and rollout: the code can land today; the staged staging→prod rollout realistically completes after today.

Separately, clean up existing orphans. The fix stops new ones, but today's stale records need a sweep — the signature is easy: any managed record whose target no longer resolves (NXDOMAIN). ab.dk is very unlikely to be the only one.

Done when removing a hostname reliably reaps its record everywhere, a regression test guards it, and the orphan sweep is clear.

[ecv]

Update from a dev-cluster repro attempt + static analysis — it refines the root cause and rules one thing out.

The label-selection theory is refuted. The reaper's List selector is a strict subset of the labels written on every record, and both were introduced in the same original commit — there's no version skew where a record could be labeled such that cleanup wouldn't see it. So "the record falls outside the reaper's set" is not it.

Empirical repro was blocked: the dev harness has no DNS substrate (DNS integration is off by default and the DNSRecordSet/DNSZone CRDs aren't installed there), so records and cleanup never run. Standing up a faithful one is harness construction, not a repro — flagging that the test suite for this needs to bring its own DNS substrate.

So the defect, if reap genuinely fails, is one of:

1. The hostname was never actually removed from the upstream proxy/gateway spec.hostnames, so it stays "desired" and cleanup correctly keeps the record — and the real problem is the canonical/binding teardown (the *.datumproxy.net target going away) rather than a reaper defect.
2. Hostname removal doesn't re-trigger the reconcile or doesn't shrink the desired-set, so cleanup never runs with the reduced set. That lives in the HTTPProxy→Gateway hostname-sync / reconcile trigger, not in the reaper itself.

Either way it's the untested create → remove-hostname round-trip, not the label logic. The test suite should target exactly that path.