Skip to content

fix: Serve DNSRecordSet webhooks on every replica - #253

Merged
vvoytovych merged 3 commits into
mainfrom
fix/dnsrecordset-webhook-replicas
Oct 18, 2026
Merged

vvoytovych merged 3 commits into
mainfrom
fix/dnsrecordset-webhook-replicas

Conversation

@vvoytovych

Copy link
Copy Markdown
Contributor

The DNSRecordSet webhooks read each record set's zone from its project, but only the elected replica connects to projects. Any other replica finds no zone and admits the record set without its display annotations.

This connects every replica to its projects, and keeps a new pod unready until its webhook server serves and its projects are connected.

Change Effect
Every replica connects to its projects, not only the elected one a replica that is not the leader answers webhook calls as the leader does
Readiness waits for the webhook server and for the projects found at start during a rollout the old pod keeps serving until the new one can answer
  • With one replica and leader election off, as deployed today, a replacement still refuses writes until the new pod is ready. That pod is now ready only once its projects are connected, about two minutes judging by network-services-operator's pods (not measured here). Running two replicas, a later step, removes the gap.
  • A replica connects to projects the way network-services-operator's does.
  • The provider is kept from starting a second time only for the leader's sake. There, a second start was measured to change nothing.

Test plan

  • Locally, the tamper test fails on main's wiring and passes with the change: a non-leader replica reads the zone from the project, and stays unready until its webhook serves
  • Locally, as deployed today (one replica, no leader election), the replicator reconciles a project's zone and the webhook annotates a record set, on main's wiring and with the change
  • Removing the connection on every replica, or either readiness check, fails a test, also with leader election off
  • CI passes on this branch

Related to #252

The replicator handed the Milo project provider to the multicluster manager
as it is, so the provider's Start ran on the elected leader only. A replica
that is not the leader engaged no project, and its mutating webhook read each
record set's DNSZone from the deployment cluster, found none, and admitted
the record set without its display annotations.

Hide Start from that wiring and start the provider on every replica, as
network-services-operator does. Gate readiness on the webhook server serving
and on the projects found at start being registered, so a new replica takes
no admission request it cannot answer.
One replica, leader election off, the Milo provider: the replicator
reconciles the project's DNSZone and the webhook annotates a record set.
The same test passes on the previous wiring, so it guards production's
configuration.
The lll linter caps a line at 120 characters. Two lines in the
non-leader test were longer, so the Lint workflow failed on this branch.
@vvoytovych
vvoytovych marked this pull request as ready for review October 18, 2026 15:02
@vvoytovych
vvoytovych requested a review from a team as a code owner October 18, 2026 15:02
@vvoytovych
vvoytovych merged commit 6a5612d into main Oct 18, 2026
12 checks passed
@vvoytovych
vvoytovych deleted the fix/dnsrecordset-webhook-replicas branch October 18, 2026 15:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants