feat(lab): Enforce traffic policy in the containerlab lab

AGENTS.md

@@ -102,7 +102,7 @@ See [docs/node-labels.md](docs/node-labels.md) for the full strategy — why `ga
 - **`config/fabric-api/`** — the fabric looking glass's cell-wide prerequisites and per-cell gateway. `kubectl apply -k config/fabric-api/` applies only the `fabric-api-gateway` and `fabric-api-janitor` ServiceAccounts and RBAC (the janitor's delete access is deliberately a separate identity from the gateway's, which can only read `FabricQuery` and write its status); safe and idempotent in any cell. `config/fabric-api/base/` (the 2-replica leader-elected gateway Deployment, its ClusterIP debug Service, a PodDisruptionBudget and the `*/15` janitor CronJob) is **not** included in that kustomization and is **not** applied as-is, for the same reason as `config/galactic-gateway/base/`: it needs per-cell values — a `fabric-api` ConfigMap (`site`, `cell`, `cluster-name`) and the cell name in its certificate's URI SAN — supplied by a per-cell overlay; see `deploy/containerlab/resources/fabric-api/gateway/` for a worked example. `config/fabric-api/crd/` is a generated `FabricQuery` CRD for the lab and tests only; infra owns the CRD (and the per-cell `fabric-api` cert-manager `Issuer`) in every cell. The node sidecar is the `config/fabric-router/components/fabric-api` Component, which also adds a `fabric-api-certs` DaemonSet (`fabric-api certsync`) that owns the cert-manager csi-driver volume and copies the certificate to the node's `/run/fabric-api/tls` for the sidecar: a csi volume blocks its pod until the certificate is issued, so keeping it out of the `fabric-router` pod means a broken Issuer stalls only the certs pod, never FRR. The overlay's `CELL_NAME` replacement applies to that DaemonSet's `fabric-api-csi` volume. Not part of the root `config/kustomization.yaml`: fabric-api is opt-in. See [docs/fabric-api/configuration.md](docs/fabric-api/configuration.md) and [docs/fabric-api/api.md](docs/fabric-api/api.md).
 - **`config/crd/`** and **`config/galactic-controller/`** — the `galactic.datumapis.com` CRDs (TrafficPolicy, TrafficPolicyBinding) and `galactic-controller --role=cell`, the TrafficPolicy validating webhook (fail closed), default-policy provisioning and binding resolution. `kubectl apply -k config/galactic-controller/` applies only its ServiceAccount and RBAC; `base/` (two-replica Deployment, webhook Service, cert-manager serving certificate, ValidatingWebhookConfiguration) is installed with the CRDs (including the cloud operator's `VPCAttachment`, which it watches) and cert-manager in place. Neither is in the root `config/kustomization.yaml`. The node side is `galactic-router` with `GALACTIC_ROUTER_TRAFFIC_POLICY=true` (off by default; `internal/trafficpolicy/node`), which publishes each attachment's bound policies into the uSID datapath, writes the binding's `status.nodes` and exports `galactic_cni_traffic_policy_*`; galactic-cni's `GALACTIC_CNI_TRAFFIC_POLICY` decides whether the datapath enforces them. See [docs/traffic-policy/controller.md](docs/traffic-policy/controller.md) and [docs/traffic-policy/node.md](docs/traffic-policy/node.md).
 - **`config/monitoring/`** — Prometheus Operator resources for BGP and NAT: a `fabric-router` PodMonitor (frr-exporter, port `frr-metrics`/`9342`, as `job=fabric-router`; fabric-config-agent, port `agent-metrics`/`9343`, as `job=fabric-config-agent`), a `galactic-router` PodMonitor covering both roles (port `metrics`/`9179`), a `galactic-nat` PodMonitor for the egress shards (port `metrics`/`9182`), each adding a `node` label, and the `galactic-bgp` PrometheusRule: underlay session down, administratively down for an hour, flapping, received prefixes halved, exporter collector or scrape failing, a session announcing an IPv4-mapped next hop, BGP routes not installed, the agent's metrics scrape down, a `galactic-router` BMP station or metrics scrape down, `galactic-router` routes failing to install, an egress shard's session table evicting live sessions, an egress shard out of masquerade ports, a stalled or persistently lossy NIC receive queue on a `galactic-nat` uplink (#673), and a `galactic-nat` metrics scrape down. For the opt-in fabric-api it adds the `fabric-router` PodMonitor's `api-metrics` endpoint (the sidecar, port `api-metrics`/`9345`, as `job=fabric-api-node`), a `fabric-api-gateway` PodMonitor (port `metrics`/`9347`) and a `fabric-api` rule group (stalled execution, node diagnostics unavailable, nodes unreachable, oversized results, cleanup backlog, certificate renewal failing, credentials missing, search index unsynced, scrape down — none fires merely because nobody ran a query). Requires the `monitoring.coreos.com` CRDs, which this repo does not install, so it is not part of the root `config/kustomization.yaml`. `task test:alerts` unit-tests the rules with `promtool` (`config/monitoring/tests/`). Every alert carries what infra's alerting standard checks for (`service: galactic`, `team: connect`, `runbook_url` into infra's `docs/runbooks/galactic/README.md`). The PrometheusRule here is the tested source, not where the rules run on Datum's infra: there the PodMonitors are converted and scraped on each edge cluster, but edge vmalert never notifies, so infra evaluates a copy of the rules centrally. Change the rules here first, with their tests, then update infra's copy.
-- **`deploy/containerlab/`** — ContainerLab topology (`gvpc.clab.yaml`) for three Kind clusters (dfw, iad, sjc) wired over an SRv6 transit mesh whose underlay is dual-stack — every transit link and loopback carries an IPv4 and an IPv6 address, with one BGP session per address family (the SRv6 data plane itself stays IPv6-only; tenant IPv4 rides inside the encapsulation). Each site has one `compute` worker and at least one `edge` worker (dfw has two); every worker has its own eBGP uplink to its site's transit router, with FRR running as a hostNetwork DaemonSet. `galactic-router` (plain mode) distributes EVPN paths over iBGP to one route reflector for the whole lab (`iad-worker3`, `config/galactic-router/overlays/control/`) — compute and edge nodes never peer directly. Every `edge` node runs an egress shard (`galactic-nat`) and a `galactic-gateway`, which share the node's uplinks through its XDP dispatcher (`GALACTIC_NAT_XDP_ATTACH=dispatch`). Each site's compute node egresses only through its own site's edge shards, and each site has its own VIP `/64`, served only through its own edge nodes to one `ns60` backend in that site. A standalone off-cluster host on the fourth transit router gives tenant egress a real destination and is the client for the VIP checks (`verify:gateway-ingress`). See `deploy/containerlab/README.md` and `deploy/containerlab/Taskfile.yaml` for bring-up commands.
+- **`deploy/containerlab/`** — ContainerLab topology (`gvpc.clab.yaml`) for three Kind clusters (dfw, iad, sjc) wired over an SRv6 transit mesh whose underlay is dual-stack — every transit link and loopback carries an IPv4 and an IPv6 address, with one BGP session per address family (the SRv6 data plane itself stays IPv6-only; tenant IPv4 rides inside the encapsulation). Each site has one `compute` worker and at least one `edge` worker (dfw has two); every worker has its own eBGP uplink to its site's transit router, with FRR running as a hostNetwork DaemonSet. `galactic-router` (plain mode) distributes EVPN paths over iBGP to one route reflector for the whole lab (`iad-worker3`, `config/galactic-router/overlays/control/`) — compute and edge nodes never peer directly. Every `edge` node runs an egress shard (`galactic-nat`) and a `galactic-gateway`, which share the node's uplinks through its XDP dispatcher (`GALACTIC_NAT_XDP_ATTACH=dispatch`). Each site's compute node egresses only through its own site's edge shards, and each site has its own VIP `/64`, served only through its own edge nodes to one `ns60` backend in that site. A standalone off-cluster host on the fourth transit router gives tenant egress a real destination and is the client for the VIP checks (`verify:gateway-ingress`). The lab enforces interface traffic policy (production defaults to off): every tenant binds each attachment to a grant set in `resources/tenants/policies/`, and `verify:traffic-policy` proves each path's grant and denial. See `deploy/containerlab/README.md` and `deploy/containerlab/Taskfile.yaml` for bring-up commands.
 
 ## New Developer Entry Points
 

api/v1alpha1/labfixtures_test.go

@@ -0,0 +1,225 @@
+// Copyright 2026 Datum Cloud, Inc.
+//
+// SPDX-License-Identifier: AGPL-3.0-or-later
+
+package v1alpha1_test
+
+import (
+	"bufio"
+	"bytes"
+	"context"
+	"encoding/json"
+	"errors"
+	"io"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+
+	"k8s.io/apimachinery/pkg/util/validation/field"
+	utilyaml "k8s.io/apimachinery/pkg/util/yaml"
+	"sigs.k8s.io/yaml"
+
+	galacticv1alpha1 "go.datum.net/galactic/api/v1alpha1"
+	"go.datum.net/galactic/internal/trafficpolicy"
+)
+
+// labTenants is the containerlab lab's tenant fixtures, which enforce traffic
+// policy (deploy/containerlab/resources/tenants/).
+var labTenants = filepath.Join("..", "..", "deploy", "containerlab", "resources", "tenants")
+
+// yamlDocs splits every YAML document in path.
+func yamlDocs(t *testing.T, path string) [][]byte {
+	t.Helper()
+	raw, err := os.ReadFile(path)
+	if err != nil {
+		t.Fatal(err)
+	}
+	r := utilyaml.NewYAMLReader(bufio.NewReader(bytes.NewReader(raw)))
+	var out [][]byte
+	for {
+		doc, err := r.Read()
+		if errors.Is(err, io.EOF) {
+			return out
+		}
+		if err != nil {
+			t.Fatalf("%s: %v", path, err)
+		}
+		if len(bytes.TrimSpace(doc)) > 0 {
+			out = append(out, doc)
+		}
+	}
+}
+
+// labPolicies reads every policy set, by set name and policy name.
+func labPolicies(t *testing.T) map[string]map[string]galacticv1alpha1.TrafficPolicy {
+	t.Helper()
+	files, err := filepath.Glob(filepath.Join(labTenants, "policies", "*", "policies.yaml"))
+	if err != nil || len(files) == 0 {
+		t.Fatalf("no lab policy sets under %s: %v", labTenants, err)
+	}
+	sets := map[string]map[string]galacticv1alpha1.TrafficPolicy{}
+	for _, f := range files {
+		set := filepath.Base(filepath.Dir(f))
+		sets[set] = map[string]galacticv1alpha1.TrafficPolicy{}
+		for _, doc := range yamlDocs(t, f) {
+			var p galacticv1alpha1.TrafficPolicy
+			if err := yaml.UnmarshalStrict(doc, &p); err != nil {
+				t.Fatalf("%s: %v", f, err)
+			}
+			sets[set][p.Name] = p
+		}
+	}
+	return sets
+}
+
+// TestLabPoliciesAreValid: every policy the lab enforces is admitted by the
+// CRD schema and passes the webhook's Go validation and the compiler, so a
+// lab deploy never stalls on a fixture the webhook refuses.
+func TestLabPoliciesAreValid(t *testing.T) {
+	sets := labPolicies(t)
+	for set, policies := range sets {
+		for name, p := range policies {
+			model, errs := trafficpolicy.FromAPI(p.Spec, field.NewPath("spec"))
+			if len(errs) > 0 {
+				t.Errorf("%s/%s: %v", set, name, errs)
+				continue
+			}
+			if _, err := trafficpolicy.Compile(model); err != nil {
+				t.Errorf("%s/%s: %v", set, name, err)
+			}
+		}
+	}
+
+	c, _ := startAPIServer(t)
+	for set, policies := range sets {
+		for name, p := range policies {
+			obj := p.DeepCopy()
+			obj.Name = set + "-" + name
+			obj.Namespace = testNamespace
+			if err := c.Create(context.Background(), obj); err != nil {
+				t.Errorf("%s/%s: the API server refused it: %v", set, name, err)
+			}
+		}
+	}
+}
+
+// TestLabTenantsAreBound: every galactic attachment a lab tenant defines has
+// a binding in that tenant's policy/, naming policies its set defines, so no
+// lab tenant comes up denying everything.
+func TestLabTenantsAreBound(t *testing.T) {
+	sets := labPolicies(t)
+	tenants, err := filepath.Glob(filepath.Join(labTenants, "ns*"))
+	if err != nil || len(tenants) == 0 {
+		t.Fatalf("no lab tenants under %s: %v", labTenants, err)
+	}
+	for _, dir := range tenants {
+		ns := filepath.Base(dir)
+		set := policySet(t, filepath.Join(dir, "policy", "kustomization.yaml"))
+		policies, ok := sets[set]
+		if !ok {
+			t.Errorf("%s: policy/ uses set %q, which does not exist", ns, set)
+			continue
+		}
+		bound := map[string]bool{}
+		for _, doc := range yamlDocs(t, filepath.Join(dir, "policy", "bindings.yaml")) {
+			var b galacticv1alpha1.TrafficPolicyBinding
+			if err := yaml.UnmarshalStrict(doc, &b); err != nil {
+				t.Fatalf("%s: %v", ns, err)
+			}
+			bound[b.Name] = true
+			for _, ref := range []*galacticv1alpha1.LocalPolicyReference{b.Spec.IngressPolicyRef, b.Spec.EgressPolicyRef} {
+				if ref == nil {
+					t.Errorf("%s/%s: a direction is unbound, so it denies everything", ns, b.Name)
+				} else if _, ok := policies[ref.Name]; !ok {
+					t.Errorf("%s/%s: names %q, which set %s does not define", ns, b.Name, ref.Name, set)
+				}
+			}
+		}
+		nads := map[string]bool{}
+		for _, nad := range galacticNADs(t, dir) {
+			nads[nad] = true
+			if !bound[nad] {
+				t.Errorf("%s: attachment %s has no binding in policy/bindings.yaml", ns, nad)
+			}
+		}
+		for name := range bound {
+			if !nads[name] {
+				t.Errorf("%s: binding %s names no attachment the tenant defines", ns, name)
+			}
+		}
+	}
+}
+
+// policySet is the policy set a tenant's policy/kustomization.yaml pulls in.
+// It fails unless the kustomization also applies bindings.yaml.
+func policySet(t *testing.T, path string) string {
+	t.Helper()
+	var k struct {
+		Resources []string `json:"resources"`
+	}
+	raw, err := os.ReadFile(path)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if err := yaml.Unmarshal(raw, &k); err != nil {
+		t.Fatalf("%s: %v", path, err)
+	}
+	set := ""
+	bindings := false
+	for _, r := range k.Resources {
+		switch {
+		case strings.Contains(r, "policies/"):
+			set = filepath.Base(r)
+		case r == "bindings.yaml":
+			bindings = true
+		}
+	}
+	if set == "" || !bindings {
+		t.Fatalf("%s must pull in one policy set and bindings.yaml, has %v", path, k.Resources)
+	}
+	return set
+}
+
+// galacticNADs names every NetworkAttachmentDefinition under a tenant's site
+// directories whose chain runs galactic-veth or galactic-tap.
+func galacticNADs(t *testing.T, dir string) []string {
+	t.Helper()
+	files, err := filepath.Glob(filepath.Join(dir, "*", "*.yaml"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	seen := map[string]bool{}
+	var out []string
+	for _, f := range files {
+		for _, doc := range yamlDocs(t, f) {
+			var nad struct {
+				Kind     string `json:"kind"`
+				Metadata struct {
+					Name string `json:"name"`
+				} `json:"metadata"`
+				Spec struct {
+					Config string `json:"config"`
+				} `json:"spec"`
+			}
+			if err := yaml.Unmarshal(doc, &nad); err != nil || nad.Kind != "NetworkAttachmentDefinition" {
+				continue
+			}
+			var conf struct {
+				Plugins []struct {
+					Type string `json:"type"`
+				} `json:"plugins"`
+			}
+			if err := json.Unmarshal([]byte(nad.Spec.Config), &conf); err != nil {
+				t.Fatalf("%s: %s: %v", f, nad.Metadata.Name, err)
+			}
+			for _, p := range conf.Plugins {
+				if (p.Type == "galactic-veth" || p.Type == "galactic-tap") && !seen[nad.Metadata.Name] {
+					seen[nad.Metadata.Name] = true
+					out = append(out, nad.Metadata.Name)
+				}
+			}
+		}
+	}
+	return out
+}