feat(router): Apply traffic policy bindings on each node

.github/workflows/ci.yaml

@@ -164,6 +164,14 @@ jobs:
         # `env "PATH=$PATH"`) already covers.
         uses: ./.github/actions/install-ebpf-deps
 
+      - name: Install envtest binaries
+        # The traffic policy node reconciler's test needs a real API server
+        # as well as root, and skips without one.
+        run: |
+          assets="$(go run sigs.k8s.io/controller-runtime/tools/setup-envtest@release-0.24 use 1.35.x --bin-dir "$RUNNER_TEMP/envtest" -p path)"
+          test -x "$assets/kube-apiserver"
+          echo "KUBEBUILDER_ASSETS=$assets" >> "$GITHUB_ENV"
+
       - name: Run root-gated unit tests as root
         # sudo's secure_path policy overrides -E's preserved PATH for
         # exactly the PATH variable, which would hide the go/task binaries

AGENTS.md

@@ -100,7 +100,7 @@ See [docs/node-labels.md](docs/node-labels.md) for the full strategy — why `ga
 - **`config/fabric-router/`** — the FRR underlay eBGP DaemonSet (`fabric-router`; `galactic-router` needs a working underlay before it can start). Unlike `config/galactic-router/`, this is a single flat DaemonSet with no `default`/`rr`-style role split — its affinity matches any node labeled `galactic.datumapis.com/fabric: router` (a dedicated mode-enum label, independent of whatever `galactic.datumapis.com/node` role value or `galactic.datumapis.com/galactic` mode that node also carries) directly, since there's no env/config difference between running on a regular node vs. those roles — fabric-router itself is identical everywhere; only the per-node BGP underlay config it reads from the ConfigMap differs. That affinity can legitimately match more than one node per cluster, and BGP underlay config (hostname, router-id, interface addresses, remote-AS) inherently differs per physical node — so each node's config lives in its own `fabric-router.<nodename>` ConfigMap (a `frr.conf` key, plus optional `daemons`/`vtysh.conf` overrides of the image defaults) rather than one shared ConfigMap for the whole DaemonSet. `fabric-config-agent` (`cmd/fabric-config-agent`, `internal/fabricconfig`, built into the `fabric-router` image) reads it from the API: as the `frr-init` init container it validates (`vtysh -C`) and installs it, holding the pod in `Init` until a valid one exists; as the `config-agent` sidecar it applies later changes live with `frr-reload.py`, so an edit to one node's ConfigMap neither restarts the pod nor touches other nodes, and reports `Applied`/`Rejected`/`ReloadFailed`/`RestartRequired` events on the ConfigMap. The sidecar also hard-resets any IPv6 session announcing an IPv4-mapped next hop (`internal/fabricconfig/nexthop.go`) and serves that check's metrics on the node's port `9343`. A third container, `frr-exporter` (upstream `tynany/frr_exporter`, pinned by digest), reads the daemons' vty sockets in the same pod-scoped `/run/frr` and serves BGP peer state, message counters and prefix counts for both underlay families on the node's port `9342` (`hostNetwork`); `task verify:fabric-metrics` in `deploy/containerlab/` checks it on every lab node. The image's default `daemons` loads bgpd's BMP module (`-M bmp`), so a node's `frr.conf` can stream to a BMP collector (`config/bmp-collector/`) with a `bmp targets` block; loading it alone streams nothing, and a change to the modules needs a pod restart rather than a reload. Every lab node's `frr.conf` streams its pre-policy Adj-RIB-In and Loc-RIB for both families, sourced from `lo` (`source-interface lo`), since the underlay carries loopbacks but not link subnets. Production `frr.conf`s come from infra's renderer, which has to emit the same block for production nodes to stream. A node with no ConfigMap of its own falls back, with a `DeprecatedConfigSource` warning, to key `frr.conf.<nodename>` of the legacy shared `fabric-config` ConfigMap. **Not** part of the root `config/kustomization.yaml` and not covered by `kubectl apply -k config/` — unlike every other component here, it has no generic default: the deployer must hand-author one `fabric-router.<nodename>` ConfigMap per matching node before applying `kubectl apply -k config/fabric-router/` (which also applies the agent's ServiceAccount and namespaced Role). The opt-in Component `config/fabric-router/components/fabric-api` adds the fabric-api sidecar to this DaemonSet's pod (see `config/fabric-api/` below); applying it, and every later fabric-api image change, recreates the pod and restarts FRR with it, so roll it out as a routing change.
 - **`config/bmp-collector/`** — a single-replica GoBMP Deployment and its `bmp-collector` Service (TCP `5000`), the BMP (RFC 7854) collector `galactic-router` and `fabric-router` stream to; `galactic-router` does so when `GALACTIC_ROUTER_BMP_STATIONS` names it (`bmp-collector.galactic-system.svc:5000`; see [docs/router/configuration.md](docs/router/configuration.md)). It parses every Peer Up/Down, route-monitoring and statistics message to JSON on stdout, so the cluster's log pipeline is its store; an overlay swapping `--dump=console` for `--kafka-server` publishes to Kafka instead. Opt-in, like `config/fabric-router/`: not part of the root `config/kustomization.yaml`, since nothing streams to it until a router is configured to. The containerlab lab runs one on `iad-worker3`'s host network (`deploy/containerlab/resources/bmp-collector/iad/`), because its three clusters cannot reach each other's Services, and points every node's `fabric-router` and the EVPN route reflector at it. Of the `galactic-router`s, only the reflector streams: its pre-policy Adj-RIB-In already holds every node's advertisements.
 - **`config/fabric-api/`** — the fabric looking glass's cell-wide prerequisites and per-cell gateway. `kubectl apply -k config/fabric-api/` applies only the `fabric-api-gateway` and `fabric-api-janitor` ServiceAccounts and RBAC (the janitor's delete access is deliberately a separate identity from the gateway's, which can only read `FabricQuery` and write its status); safe and idempotent in any cell. `config/fabric-api/base/` (the 2-replica leader-elected gateway Deployment, its ClusterIP debug Service, a PodDisruptionBudget and the `*/15` janitor CronJob) is **not** included in that kustomization and is **not** applied as-is, for the same reason as `config/galactic-gateway/base/`: it needs per-cell values — a `fabric-api` ConfigMap (`site`, `cell`, `cluster-name`) and the cell name in its certificate's URI SAN — supplied by a per-cell overlay; see `deploy/containerlab/resources/fabric-api/gateway/` for a worked example. `config/fabric-api/crd/` is a generated `FabricQuery` CRD for the lab and tests only; infra owns the CRD (and the per-cell `fabric-api` cert-manager `Issuer`) in every cell. The node sidecar is the `config/fabric-router/components/fabric-api` Component, which also adds a `fabric-api-certs` DaemonSet (`fabric-api certsync`) that owns the cert-manager csi-driver volume and copies the certificate to the node's `/run/fabric-api/tls` for the sidecar: a csi volume blocks its pod until the certificate is issued, so keeping it out of the `fabric-router` pod means a broken Issuer stalls only the certs pod, never FRR. The overlay's `CELL_NAME` replacement applies to that DaemonSet's `fabric-api-csi` volume. Not part of the root `config/kustomization.yaml`: fabric-api is opt-in. See [docs/fabric-api/configuration.md](docs/fabric-api/configuration.md) and [docs/fabric-api/api.md](docs/fabric-api/api.md).
-- **`config/crd/`** and **`config/galactic-controller/`** — the `galactic.datumapis.com` CRDs (TrafficPolicy, TrafficPolicyBinding) and `galactic-controller --role=cell`, the TrafficPolicy validating webhook (fail closed), default-policy provisioning and binding resolution. `kubectl apply -k config/galactic-controller/` applies only its ServiceAccount and RBAC; `base/` (two-replica Deployment, webhook Service, cert-manager serving certificate, ValidatingWebhookConfiguration) is installed with the CRDs (including the cloud operator's `VPCAttachment`, which it watches) and cert-manager in place. Neither is in the root `config/kustomization.yaml`: nothing enforces traffic policy yet. See [docs/traffic-policy/controller.md](docs/traffic-policy/controller.md).
+- **`config/crd/`** and **`config/galactic-controller/`** — the `galactic.datumapis.com` CRDs (TrafficPolicy, TrafficPolicyBinding) and `galactic-controller --role=cell`, the TrafficPolicy validating webhook (fail closed), default-policy provisioning and binding resolution. `kubectl apply -k config/galactic-controller/` applies only its ServiceAccount and RBAC; `base/` (two-replica Deployment, webhook Service, cert-manager serving certificate, ValidatingWebhookConfiguration) is installed with the CRDs (including the cloud operator's `VPCAttachment`, which it watches) and cert-manager in place. Neither is in the root `config/kustomization.yaml`. The node side is `galactic-router` with `GALACTIC_ROUTER_TRAFFIC_POLICY=true` (off by default; `internal/trafficpolicy/node`), which publishes each attachment's bound policies into the uSID datapath, writes the binding's `status.nodes` and exports `galactic_cni_traffic_policy_*`; galactic-cni's `GALACTIC_CNI_TRAFFIC_POLICY` decides whether the datapath enforces them. See [docs/traffic-policy/controller.md](docs/traffic-policy/controller.md) and [docs/traffic-policy/node.md](docs/traffic-policy/node.md).
 - **`config/monitoring/`** — Prometheus Operator resources for BGP and NAT: a `fabric-router` PodMonitor (frr-exporter, port `frr-metrics`/`9342`, as `job=fabric-router`; fabric-config-agent, port `agent-metrics`/`9343`, as `job=fabric-config-agent`), a `galactic-router` PodMonitor covering both roles (port `metrics`/`9179`), a `galactic-nat` PodMonitor for the egress shards (port `metrics`/`9182`), each adding a `node` label, and the `galactic-bgp` PrometheusRule: underlay session down, administratively down for an hour, flapping, received prefixes halved, exporter collector or scrape failing, a session announcing an IPv4-mapped next hop, BGP routes not installed, the agent's metrics scrape down, a `galactic-router` BMP station or metrics scrape down, `galactic-router` routes failing to install, an egress shard's session table evicting live sessions, an egress shard out of masquerade ports, a stalled or persistently lossy NIC receive queue on a `galactic-nat` uplink (#673), and a `galactic-nat` metrics scrape down. For the opt-in fabric-api it adds the `fabric-router` PodMonitor's `api-metrics` endpoint (the sidecar, port `api-metrics`/`9345`, as `job=fabric-api-node`), a `fabric-api-gateway` PodMonitor (port `metrics`/`9347`) and a `fabric-api` rule group (stalled execution, node diagnostics unavailable, nodes unreachable, oversized results, cleanup backlog, certificate renewal failing, credentials missing, search index unsynced, scrape down — none fires merely because nobody ran a query). Requires the `monitoring.coreos.com` CRDs, which this repo does not install, so it is not part of the root `config/kustomization.yaml`. `task test:alerts` unit-tests the rules with `promtool` (`config/monitoring/tests/`). Every alert carries what infra's alerting standard checks for (`service: galactic`, `team: connect`, `runbook_url` into infra's `docs/runbooks/galactic/README.md`). The PrometheusRule here is the tested source, not where the rules run on Datum's infra: there the PodMonitors are converted and scraped on each edge cluster, but edge vmalert never notifies, so infra evaluates a copy of the rules centrally. Change the rules here first, with their tests, then update infra's copy.
 - **`deploy/containerlab/`** — ContainerLab topology (`gvpc.clab.yaml`) for three Kind clusters (dfw, iad, sjc) wired over an SRv6 transit mesh whose underlay is dual-stack — every transit link and loopback carries an IPv4 and an IPv6 address, with one BGP session per address family (the SRv6 data plane itself stays IPv6-only; tenant IPv4 rides inside the encapsulation). Each site has one `compute` worker and at least one `edge` worker (dfw has two); every worker has its own eBGP uplink to its site's transit router, with FRR running as a hostNetwork DaemonSet. `galactic-router` (plain mode) distributes EVPN paths over iBGP to one route reflector for the whole lab (`iad-worker3`, `config/galactic-router/overlays/control/`) — compute and edge nodes never peer directly. Every `edge` node runs an egress shard (`galactic-nat`) and a `galactic-gateway`, which share the node's uplinks through its XDP dispatcher (`GALACTIC_NAT_XDP_ATTACH=dispatch`). Each site's compute node egresses only through its own site's edge shards, and each site has its own VIP `/64`, served only through its own edge nodes to one `ns60` backend in that site. A standalone off-cluster host on the fourth transit router gives tenant egress a real destination and is the client for the VIP checks (`verify:gateway-ingress`). See `deploy/containerlab/README.md` and `deploy/containerlab/Taskfile.yaml` for bring-up commands.
 

cmd/galactic-router/main.go

@@ -14,6 +14,8 @@ import (
 	authorizationv1 "k8s.io/api/authorization/v1"
 	ctrl "sigs.k8s.io/controller-runtime"
 	"sigs.k8s.io/controller-runtime/pkg/client"
+
+	galacticv1alpha1 "go.datum.net/galactic/api/v1alpha1"
 )
 
 const (
@@ -45,7 +47,7 @@ func main() {
 // resource the manager watches. A denial means the informer cache never syncs
 // and every reconciler is silently blocked, so this logs a clear, actionable
 // message at startup instead.
-func checkWatchPermissions(mgr ctrl.Manager) {
+func checkWatchPermissions(mgr ctrl.Manager, trafficPolicy bool) {
 	c, err := client.New(mgr.GetConfig(), client.Options{Scheme: mgr.GetScheme()})
 	if err != nil {
 		ctrl.Log.Error(err, "RBAC pre-flight: cannot create client, skipping check")
@@ -76,6 +78,17 @@ func checkWatchPermissions(mgr ctrl.Manager) {
 		{version: "v1", resource: resourceSecrets},
 		{version: "v1", resource: resourceNodes},
 	}
+	if trafficPolicy {
+		resources = append(resources,
+			struct{ group, version, resource, namespace string }{
+				group: galacticv1alpha1.GroupVersion.Group, version: galacticv1alpha1.GroupVersion.Version,
+				resource: "trafficpolicies",
+			},
+			struct{ group, version, resource, namespace string }{
+				group: galacticv1alpha1.GroupVersion.Group, version: galacticv1alpha1.GroupVersion.Version,
+				resource: "trafficpolicybindings",
+			})
+	}
 
 	for _, r := range resources {
 		review := &authorizationv1.SelfSubjectAccessReview{