unbounded-stable: Site/cluster node CIDR does not match any node, and boulderlab blocks the release gate

[plombardi89]

Summary

unbounded-stable has not completed a soak since 2026-08-12. Two separate
problems, both currently blocking releases from deploying there.

1. Site/cluster claims a node CIDR that matches nothing

Site/cluster    spec.nodeCidrs: ["10.200.0.0/16"]
AKS nodes:      10.224.0.4 .. 10.224.0.11

Disjoint ranges. Every AKS node carries no unbounded-cloud.io/* label and
belongs to no Site, because assignment matches InternalIP against
spec.nodeCidrs (internal/net/controller/site_controller.go:2534). Nodes that
belong to no Site get no pod CIDR assignment from computePodCIDRsForNode.

The other Sites are consistent: stable is 10.1.0.0/16 with its node at
10.1.0.10, boulderlab is 172.18.10.0/24 with nodes at 172.18.10.1-7.

Notably the deploy workflow's own CLUSTER_NODE_CIDR environment variable holds
the correct 10.224.0.0/12, visible in the v0.2.4 run. The Site and the
workflow are two independently maintained sources of truth and they disagree.

managedFields shows kubectl-unbounded applied f:nodeCidrs at
2026-08-17T19:16:26Z. No CI ran that day until 23:58, so this was a manual
site init/install with --cluster-node-cidr 10.200.0.0/16. The value appears
nowhere in this repository, so it came from another cluster's config or a stale
runbook. There is no audit trail: the cluster has no diagnostic settings at
all, so kube-audit was never enabled.

The product gaps that let this happen silently are filed separately as #649.

Fix: re-run the CLI with --cluster-node-cidr 10.224.0.0/12. It owns the
field via server-side apply, so patching the object directly would be undone by
the next apply.

2. metalman-controller-boulderlab cannot schedule

This is what actually fails the release gate:

deployment.apps/metalman-controller-boulderlab   0/1   ghcr.io/azure/metalman:v0.2.4
pod/...-56c85d7666-vwpbg   0/1  Pending      5m13s  <none>
pod/...-7dbc7765f-lv4tb    1/1  Terminating  4d23h  spark-3d37
NotTriggerScaleUp: pod didn't trigger scale-up: 1 node(s) didn't match Pod's node affinity/selector

The new pod cannot be scheduled because no node matches its affinity; metalman is
per-Site and pinned via SiteNodeAffinity. Its predecessor had been stuck
Terminating on spark-3d37 for nearly five days, which is the signature of a
node the control plane has lost contact with. All seven boulderlab nodes are
currently Ready=Unknown, and seven of eight AKS nodes are Ready=False.

This is not something #619 addressed. That work tolerates a DaemonSet
shortfall caused only by NotReady nodes (RELEASING.md, Degraded clusters). A
Deployment whose pod is Pending because nothing matches its affinity is neither,
so the gate failed correctly.

It will recur on every release until boulderlab is fixed, because
metalman-targets.sh discovers per-Site metalman deployments from Site spec and
adds them to the gate.

Options: restore a boulderlab node matching the affinity; or set
spec.components.metalman.enabled: false on the boulderlab Site, which removes
it from the gate cleanly since targets are discovered rather than assumed; or
force-publish each release with a recorded reason.

Worth doing regardless

Enable kube-audit-admin on this cluster. It excludes read-only calls, so volume
and cost are a fraction of full kube-audit, and it would have answered "who
changed this" in one query. Right now there is no record of any mutation anyone
has made to this cluster.