diff --git a/AGENTS.md b/AGENTS.md index a2577f4e..b4d24d91 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -102,7 +102,7 @@ See [docs/node-labels.md](docs/node-labels.md) for the full strategy — why `ga - **`config/fabric-api/`** — the fabric looking glass's cell-wide prerequisites and per-cell gateway. `kubectl apply -k config/fabric-api/` applies only the `fabric-api-gateway` and `fabric-api-janitor` ServiceAccounts and RBAC (the janitor's delete access is deliberately a separate identity from the gateway's, which can only read `FabricQuery` and write its status); safe and idempotent in any cell. `config/fabric-api/base/` (the 2-replica leader-elected gateway Deployment, its ClusterIP debug Service, a PodDisruptionBudget and the `*/15` janitor CronJob) is **not** included in that kustomization and is **not** applied as-is, for the same reason as `config/galactic-gateway/base/`: it needs per-cell values — a `fabric-api` ConfigMap (`site`, `cell`, `cluster-name`) and the cell name in its certificate's URI SAN — supplied by a per-cell overlay; see `deploy/containerlab/resources/fabric-api/gateway/` for a worked example. `config/fabric-api/crd/` is a generated `FabricQuery` CRD for the lab and tests only; infra owns the CRD (and the per-cell `fabric-api` cert-manager `Issuer`) in every cell. The node sidecar is the `config/fabric-router/components/fabric-api` Component, which also adds a `fabric-api-certs` DaemonSet (`fabric-api certsync`) that owns the cert-manager csi-driver volume and copies the certificate to the node's `/run/fabric-api/tls` for the sidecar: a csi volume blocks its pod until the certificate is issued, so keeping it out of the `fabric-router` pod means a broken Issuer stalls only the certs pod, never FRR. The overlay's `CELL_NAME` replacement applies to that DaemonSet's `fabric-api-csi` volume. Not part of the root `config/kustomization.yaml`: fabric-api is opt-in. See [docs/fabric-api/configuration.md](docs/fabric-api/configuration.md) and [docs/fabric-api/api.md](docs/fabric-api/api.md). - **`config/crd/`** and **`config/galactic-controller/`** — the `galactic.datumapis.com` CRDs (TrafficPolicy, TrafficPolicyBinding) and `galactic-controller --role=cell`, the TrafficPolicy validating webhook (fail closed), default-policy provisioning and binding resolution. `kubectl apply -k config/galactic-controller/` applies only its ServiceAccount and RBAC; `base/` (two-replica Deployment, webhook Service, cert-manager serving certificate, ValidatingWebhookConfiguration) is installed with the CRDs (including the cloud operator's `VPCAttachment`, which it watches) and cert-manager in place. Neither is in the root `config/kustomization.yaml`. The node side is `galactic-router` with `GALACTIC_ROUTER_TRAFFIC_POLICY=true` (off by default; `internal/trafficpolicy/node`), which publishes each attachment's bound policies into the uSID datapath, writes the binding's `status.nodes` and exports `galactic_cni_traffic_policy_*`; galactic-cni's `GALACTIC_CNI_TRAFFIC_POLICY` decides whether the datapath enforces them. See [docs/traffic-policy/controller.md](docs/traffic-policy/controller.md) and [docs/traffic-policy/node.md](docs/traffic-policy/node.md). - **`config/monitoring/`** — Prometheus Operator resources for BGP and NAT: a `fabric-router` PodMonitor (frr-exporter, port `frr-metrics`/`9342`, as `job=fabric-router`; fabric-config-agent, port `agent-metrics`/`9343`, as `job=fabric-config-agent`), a `galactic-router` PodMonitor covering both roles (port `metrics`/`9179`), a `galactic-nat` PodMonitor for the egress shards (port `metrics`/`9182`), each adding a `node` label, and the `galactic-bgp` PrometheusRule: underlay session down, administratively down for an hour, flapping, received prefixes halved, exporter collector or scrape failing, a session announcing an IPv4-mapped next hop, BGP routes not installed, the agent's metrics scrape down, a `galactic-router` BMP station or metrics scrape down, `galactic-router` routes failing to install, an egress shard's session table evicting live sessions, an egress shard out of masquerade ports, a stalled or persistently lossy NIC receive queue on a `galactic-nat` uplink (#673), and a `galactic-nat` metrics scrape down. For the opt-in fabric-api it adds the `fabric-router` PodMonitor's `api-metrics` endpoint (the sidecar, port `api-metrics`/`9345`, as `job=fabric-api-node`), a `fabric-api-gateway` PodMonitor (port `metrics`/`9347`) and a `fabric-api` rule group (stalled execution, node diagnostics unavailable, nodes unreachable, oversized results, cleanup backlog, certificate renewal failing, credentials missing, search index unsynced, scrape down — none fires merely because nobody ran a query). Requires the `monitoring.coreos.com` CRDs, which this repo does not install, so it is not part of the root `config/kustomization.yaml`. `task test:alerts` unit-tests the rules with `promtool` (`config/monitoring/tests/`). Every alert carries what infra's alerting standard checks for (`service: galactic`, `team: connect`, `runbook_url` into infra's `docs/runbooks/galactic/README.md`). The PrometheusRule here is the tested source, not where the rules run on Datum's infra: there the PodMonitors are converted and scraped on each edge cluster, but edge vmalert never notifies, so infra evaluates a copy of the rules centrally. Change the rules here first, with their tests, then update infra's copy. -- **`deploy/containerlab/`** — ContainerLab topology (`gvpc.clab.yaml`) for three Kind clusters (dfw, iad, sjc) wired over an SRv6 transit mesh whose underlay is dual-stack — every transit link and loopback carries an IPv4 and an IPv6 address, with one BGP session per address family (the SRv6 data plane itself stays IPv6-only; tenant IPv4 rides inside the encapsulation). Each site has one `compute` worker and at least one `edge` worker (dfw has two); every worker has its own eBGP uplink to its site's transit router, with FRR running as a hostNetwork DaemonSet. `galactic-router` (plain mode) distributes EVPN paths over iBGP to one route reflector for the whole lab (`iad-worker3`, `config/galactic-router/overlays/control/`) — compute and edge nodes never peer directly. Every `edge` node runs an egress shard (`galactic-nat`) and a `galactic-gateway`, which share the node's uplinks through its XDP dispatcher (`GALACTIC_NAT_XDP_ATTACH=dispatch`). Each site's compute node egresses only through its own site's edge shards, and each site has its own VIP `/64`, served only through its own edge nodes to one `ns60` backend in that site. A standalone off-cluster host on the fourth transit router gives tenant egress a real destination and is the client for the VIP checks (`verify:gateway-ingress`). See `deploy/containerlab/README.md` and `deploy/containerlab/Taskfile.yaml` for bring-up commands. +- **`deploy/containerlab/`** — ContainerLab topology (`gvpc.clab.yaml`) for three Kind clusters (dfw, iad, sjc) wired over an SRv6 transit mesh whose underlay is dual-stack — every transit link and loopback carries an IPv4 and an IPv6 address, with one BGP session per address family (the SRv6 data plane itself stays IPv6-only; tenant IPv4 rides inside the encapsulation). Each site has one `compute` worker and at least one `edge` worker (dfw has two); every worker has its own eBGP uplink to its site's transit router, with FRR running as a hostNetwork DaemonSet. `galactic-router` (plain mode) distributes EVPN paths over iBGP to one route reflector for the whole lab (`iad-worker3`, `config/galactic-router/overlays/control/`) — compute and edge nodes never peer directly. Every `edge` node runs an egress shard (`galactic-nat`) and a `galactic-gateway`, which share the node's uplinks through its XDP dispatcher (`GALACTIC_NAT_XDP_ATTACH=dispatch`). Each site's compute node egresses only through its own site's edge shards, and each site has its own VIP `/64`, served only through its own edge nodes to one `ns60` backend in that site. A standalone off-cluster host on the fourth transit router gives tenant egress a real destination and is the client for the VIP checks (`verify:gateway-ingress`). The lab enforces interface traffic policy (production defaults to off): every tenant binds each attachment to a grant set in `resources/tenants/policies/`, and `verify:traffic-policy` proves each path's grant and denial. See `deploy/containerlab/README.md` and `deploy/containerlab/Taskfile.yaml` for bring-up commands. ## New Developer Entry Points diff --git a/api/v1alpha1/labfixtures_test.go b/api/v1alpha1/labfixtures_test.go new file mode 100644 index 00000000..b86bdba9 --- /dev/null +++ b/api/v1alpha1/labfixtures_test.go @@ -0,0 +1,225 @@ +// Copyright 2026 Datum Cloud, Inc. +// +// SPDX-License-Identifier: AGPL-3.0-or-later + +package v1alpha1_test + +import ( + "bufio" + "bytes" + "context" + "encoding/json" + "errors" + "io" + "os" + "path/filepath" + "strings" + "testing" + + "k8s.io/apimachinery/pkg/util/validation/field" + utilyaml "k8s.io/apimachinery/pkg/util/yaml" + "sigs.k8s.io/yaml" + + galacticv1alpha1 "go.datum.net/galactic/api/v1alpha1" + "go.datum.net/galactic/internal/trafficpolicy" +) + +// labTenants is the containerlab lab's tenant fixtures, which enforce traffic +// policy (deploy/containerlab/resources/tenants/). +var labTenants = filepath.Join("..", "..", "deploy", "containerlab", "resources", "tenants") + +// yamlDocs splits every YAML document in path. +func yamlDocs(t *testing.T, path string) [][]byte { + t.Helper() + raw, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + r := utilyaml.NewYAMLReader(bufio.NewReader(bytes.NewReader(raw))) + var out [][]byte + for { + doc, err := r.Read() + if errors.Is(err, io.EOF) { + return out + } + if err != nil { + t.Fatalf("%s: %v", path, err) + } + if len(bytes.TrimSpace(doc)) > 0 { + out = append(out, doc) + } + } +} + +// labPolicies reads every policy set, by set name and policy name. +func labPolicies(t *testing.T) map[string]map[string]galacticv1alpha1.TrafficPolicy { + t.Helper() + files, err := filepath.Glob(filepath.Join(labTenants, "policies", "*", "policies.yaml")) + if err != nil || len(files) == 0 { + t.Fatalf("no lab policy sets under %s: %v", labTenants, err) + } + sets := map[string]map[string]galacticv1alpha1.TrafficPolicy{} + for _, f := range files { + set := filepath.Base(filepath.Dir(f)) + sets[set] = map[string]galacticv1alpha1.TrafficPolicy{} + for _, doc := range yamlDocs(t, f) { + var p galacticv1alpha1.TrafficPolicy + if err := yaml.UnmarshalStrict(doc, &p); err != nil { + t.Fatalf("%s: %v", f, err) + } + sets[set][p.Name] = p + } + } + return sets +} + +// TestLabPoliciesAreValid: every policy the lab enforces is admitted by the +// CRD schema and passes the webhook's Go validation and the compiler, so a +// lab deploy never stalls on a fixture the webhook refuses. +func TestLabPoliciesAreValid(t *testing.T) { + sets := labPolicies(t) + for set, policies := range sets { + for name, p := range policies { + model, errs := trafficpolicy.FromAPI(p.Spec, field.NewPath("spec")) + if len(errs) > 0 { + t.Errorf("%s/%s: %v", set, name, errs) + continue + } + if _, err := trafficpolicy.Compile(model); err != nil { + t.Errorf("%s/%s: %v", set, name, err) + } + } + } + + c, _ := startAPIServer(t) + for set, policies := range sets { + for name, p := range policies { + obj := p.DeepCopy() + obj.Name = set + "-" + name + obj.Namespace = testNamespace + if err := c.Create(context.Background(), obj); err != nil { + t.Errorf("%s/%s: the API server refused it: %v", set, name, err) + } + } + } +} + +// TestLabTenantsAreBound: every galactic attachment a lab tenant defines has +// a binding in that tenant's policy/, naming policies its set defines, so no +// lab tenant comes up denying everything. +func TestLabTenantsAreBound(t *testing.T) { + sets := labPolicies(t) + tenants, err := filepath.Glob(filepath.Join(labTenants, "ns*")) + if err != nil || len(tenants) == 0 { + t.Fatalf("no lab tenants under %s: %v", labTenants, err) + } + for _, dir := range tenants { + ns := filepath.Base(dir) + set := policySet(t, filepath.Join(dir, "policy", "kustomization.yaml")) + policies, ok := sets[set] + if !ok { + t.Errorf("%s: policy/ uses set %q, which does not exist", ns, set) + continue + } + bound := map[string]bool{} + for _, doc := range yamlDocs(t, filepath.Join(dir, "policy", "bindings.yaml")) { + var b galacticv1alpha1.TrafficPolicyBinding + if err := yaml.UnmarshalStrict(doc, &b); err != nil { + t.Fatalf("%s: %v", ns, err) + } + bound[b.Name] = true + for _, ref := range []*galacticv1alpha1.LocalPolicyReference{b.Spec.IngressPolicyRef, b.Spec.EgressPolicyRef} { + if ref == nil { + t.Errorf("%s/%s: a direction is unbound, so it denies everything", ns, b.Name) + } else if _, ok := policies[ref.Name]; !ok { + t.Errorf("%s/%s: names %q, which set %s does not define", ns, b.Name, ref.Name, set) + } + } + } + nads := map[string]bool{} + for _, nad := range galacticNADs(t, dir) { + nads[nad] = true + if !bound[nad] { + t.Errorf("%s: attachment %s has no binding in policy/bindings.yaml", ns, nad) + } + } + for name := range bound { + if !nads[name] { + t.Errorf("%s: binding %s names no attachment the tenant defines", ns, name) + } + } + } +} + +// policySet is the policy set a tenant's policy/kustomization.yaml pulls in. +// It fails unless the kustomization also applies bindings.yaml. +func policySet(t *testing.T, path string) string { + t.Helper() + var k struct { + Resources []string `json:"resources"` + } + raw, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + if err := yaml.Unmarshal(raw, &k); err != nil { + t.Fatalf("%s: %v", path, err) + } + set := "" + bindings := false + for _, r := range k.Resources { + switch { + case strings.Contains(r, "policies/"): + set = filepath.Base(r) + case r == "bindings.yaml": + bindings = true + } + } + if set == "" || !bindings { + t.Fatalf("%s must pull in one policy set and bindings.yaml, has %v", path, k.Resources) + } + return set +} + +// galacticNADs names every NetworkAttachmentDefinition under a tenant's site +// directories whose chain runs galactic-veth or galactic-tap. +func galacticNADs(t *testing.T, dir string) []string { + t.Helper() + files, err := filepath.Glob(filepath.Join(dir, "*", "*.yaml")) + if err != nil { + t.Fatal(err) + } + seen := map[string]bool{} + var out []string + for _, f := range files { + for _, doc := range yamlDocs(t, f) { + var nad struct { + Kind string `json:"kind"` + Metadata struct { + Name string `json:"name"` + } `json:"metadata"` + Spec struct { + Config string `json:"config"` + } `json:"spec"` + } + if err := yaml.Unmarshal(doc, &nad); err != nil || nad.Kind != "NetworkAttachmentDefinition" { + continue + } + var conf struct { + Plugins []struct { + Type string `json:"type"` + } `json:"plugins"` + } + if err := json.Unmarshal([]byte(nad.Spec.Config), &conf); err != nil { + t.Fatalf("%s: %s: %v", f, nad.Metadata.Name, err) + } + for _, p := range conf.Plugins { + if (p.Type == "galactic-veth" || p.Type == "galactic-tap") && !seen[nad.Metadata.Name] { + seen[nad.Metadata.Name] = true + out = append(out, nad.Metadata.Name) + } + } + } + } + return out +} diff --git a/deploy/containerlab/AGENTS.md b/deploy/containerlab/AGENTS.md index 68b8d7c6..2ab1064f 100644 --- a/deploy/containerlab/AGENTS.md +++ b/deploy/containerlab/AGENTS.md @@ -27,6 +27,8 @@ - **FRR config**: Transit router configs omit the `frr version` directive (managed by the FRR image, not the config). - **Shared manifests**: `resources/galactic-cni/kustomization.yaml` builds on `config/galactic-cni/`, `resources/galactic-router/base/kustomization.yaml` builds on `config/galactic-router/overlays/router/`, `resources/galactic-control/iad/kustomization.yaml` builds on `config/galactic-router/overlays/control/`, and `resources/fabric-router/base/kustomization.yaml` builds on `config/fabric-router/` (each in turn pulls in its own `base/` where one exists: `config/galactic-router/base/`; `config/fabric-router/` has no role split, it's a single flat DaemonSet) rather than forking them, patching in only what the lab needs to differ (image, env — for fabric, just `image: fabric-router:latest` + `imagePullPolicy: Never` on all three containers via `fabric-lab-patch.yaml`, which overrides nothing else). `config/fabric-router/`'s own affinity already covers every fabric-needing node via `galactic.datumapis.com/fabric: router` (every lab worker carries it, regardless of its other labels), so the lab inherits it as-is: one fabric DaemonSet per cluster, one `fabric-router.` ConfigMap per matching node. (A second overlay, `resources/fabric-control/iad/`, used to exist purely so the reflector node could get its own `frr.conf` back when the DaemonSet read a single shared `frr.conf` — it narrowed the base affinity to `node: [compute, edge]`, narrowed its own to `galactic: control`, and needed a JSON6902 `/metadata/name` rename to `fabric-control` to avoid clobbering the base DaemonSet in the same namespace. Per-node config selection removed the need for all three, and it was deleted; the reflector node is now served by the single DaemonSet like every other node. It was never the underlay's route reflector — that's `fabric-router-rr` on `fabric=control`, still unimplemented, see `docs/node-labels.md`.) `kubectl apply -k` refuses to load resource files from outside a kustomization's own root, so `deploy-cni.sh`/`deploy-galactic-router.sh`/`deploy-fabric.sh` `docker cp` those directories into a local `base/` subdirectory on the node at deploy time (`resources/galactic-cni/base/`, `resources/galactic-router/base/{base,overlays/router}/`, `resources/galactic-control/iad/{base,overlays/control}/` — each nested at the same relative depth as `config/galactic-router/` itself, since the copied `overlays/router`/`overlays/control` kustomization.yamls' own `../../base` references are copied verbatim, `resources/fabric-router/base/fabric/`) instead of referencing `config/` across that boundary. The `galactic-system` namespace and RBAC/ServiceAccount are applied straight from `config/galactic-system/namespace.yaml` and `config/*/{rbac,serviceaccount}.yaml` by `deploy-system.sh` via `lib.sh`'s `copy_config` (which copies all of `config/` to `/galactic/config/`) — not part of any kustomize build, so per-site `namePrefix` never touches the shared cluster-scoped RBAC. `config/galactic-system/namespace.yaml` is also the only place `galactic-system` gets created for production — there is no other bootstrap step in this repo, so if it's ever missing on a real cluster, deploying `config/galactic-router/` or `config/galactic-cni/` fails outright. +- **Traffic policy is enforced**: `resources/galactic-cni/shared/` sets `GALACTIC_CNI_TRAFFIC_POLICY=enforce` and `resources/galactic-router/base/` sets `GALACTIC_ROUTER_TRAFFIC_POLICY=true` (production defaults to off for both), so every tenant interface denies everything until galactic-router publishes its bindings. `deploy:traffic-policy` (after `deploy:cert-manager`, before `deploy:galactic-router`) installs the `galactic.datumapis.com` CRDs and `galactic-controller` (`resources/galactic-controller/`, on the host network for the same reason as cert-manager's webhook, ports 10261-10263, one replica). Each `deploy:nsNN` applies the tenant's `policy/` (one of the sets in `resources/tenants/policies/`, bound to every attachment in both directions), then its pods, then `scripts/publish-attachments.sh`, which creates any VPCAttachment the tenant does not define and writes every attachment's status, including `hostInterface` (`GH`), which the router binds by. A new tenant, NAD or check that sends a new kind of traffic needs a grant: `api/v1alpha1`'s `TestLabTenantsAreBound` fails for an unbound NAD, and `verify:traffic-policy` covers each path's grant and denial. + ## Naming Layers - **Fabric** — FRR DaemonSets running on each cluster's workers; handle eBGP to the transit mesh. Resources live in `resources/fabric-router/`. @@ -35,7 +37,7 @@ - **iad-control** — `iad-worker3`, the lab's single EVPN route reflector; every galactic-router in every cluster peers with it. galactic-router + BGP CRDs in `resources/galactic-control/iad/` (one `bgppeer-.yaml` per client), fabric in `resources/fabric-router/iad/` (`frr.conf.iad-worker3`). - **Edge** — the four `nat=enabled` workers (`dfw-worker2`, `dfw-worker3`, `sjc-worker2`, `iad-worker2`): each site's transit-facing nodes, the uplinks its compute node routes through, the egress shards and the VIP gateways. Their `BGPRouter` and route-reflector `BGPPeer` live alongside the compute node's in `resources/galactic-router//` (`bgprouter-.yaml`, `bgppeer-.yaml`). - **Egress** — `galactic-nat`, one egress shard per edge node, running from that node's XDP dispatcher on `bond0`/`bond1`. One per-site overlay under `resources/galactic-nat//` (uplink override + one `EgressShard` per edge node; dfw has two). Each site's compute node lists only its own site's shard SIDs (`resources/galactic-cni//egress-shards-patch.yaml`), and each edge node's `frr.conf` originates its shard's SID `/64` and masquerade addresses. -- **Ingress** — `galactic-gateway`, one DaemonSet per edge node (`resources/galactic-gateway//`: hostname pin, `PUBLIC_INTERFACE=bond0`, `INTERNAL_INTERFACES=bond1`, `SRV6_ADDRESS` left unset so the gateway derives the node's own uSID from its `BGPRouter`, and its `NetworkGateway`). Each site directory (`resources/galactic-gateway//`) adds the `ns60-tcp`/`ns60-udp` `NetworkRule`s for that site's own VIP, which select the site's two `ns60` backends by label through their `VPCAttachment`s (`resources/tenants/ns60//vpcattachments.yaml`, fixed addresses from their NADs; `scripts/publish-ns60-attachments.sh` writes their status, the attachment controller's job in production). The compute node's `galactic-router` writes one `ServiceVIPBinding` per rule per backend; none is hand-written. Each edge node's `frr.conf` originates its site's VIP `/64` (`2001:db8:6060:{1,2,3}::/64` for dfw, sjc, iad) to `Null0`. Both backends share the site's one compute node on purpose: the gateway carries Maglev's choice to the node in the SRv6 destination's slot, so each gets its own flows (#799). +- **Ingress** — `galactic-gateway`, one DaemonSet per edge node (`resources/galactic-gateway//`: hostname pin, `PUBLIC_INTERFACE=bond0`, `INTERNAL_INTERFACES=bond1`, `SRV6_ADDRESS` left unset so the gateway derives the node's own uSID from its `BGPRouter`, and its `NetworkGateway`). Each site directory (`resources/galactic-gateway//`) adds the `ns60-tcp`/`ns60-udp` `NetworkRule`s for that site's own VIP, which select the site's two `ns60` backends by label through their `VPCAttachment`s (`resources/tenants/ns60//vpcattachments.yaml`, fixed addresses from their NADs; `scripts/publish-attachments.sh` writes their status, as it does every tenant's, the attachment controller's job in production). The compute node's `galactic-router` writes one `ServiceVIPBinding` per rule per backend; none is hand-written. Each edge node's `frr.conf` originates its site's VIP `/64` (`2001:db8:6060:{1,2,3}::/64` for dfw, sjc, iad) to `Null0`. Both backends share the site's one compute node on purpose: the gateway carries Maglev's choice to the node in the SRv6 destination's slot, so each gets its own flows (#799). ## References diff --git a/deploy/containerlab/README.md b/deploy/containerlab/README.md index aa3c0453..716951eb 100644 --- a/deploy/containerlab/README.md +++ b/deploy/containerlab/README.md @@ -418,6 +418,7 @@ deploy/containerlab/ │ ├── galactic-nat/ # egress shards per site, one per edge node (dfw has two) │ ├── galactic-gateway/ # one gateway overlay per edge node, plus each site's │ │ # ns60 NetworkRules (backends picked by selector) +│ ├── galactic-controller/ # traffic policy's cell controller, on the host network │ └── tenants/ # test VPCs — one shared base/ (Namespace + netshoot │ ├── base/ # Deployment), each tenant patching its namespace and │ ├── ns10/ # default-network annotation; per-site dirs hold each @@ -428,9 +429,11 @@ deploy/containerlab/ │ │ # their own NAD+Deployment (distinct vpcattachment, │ │ # same vpc), not one NAD scaled to replicas: 2 — see │ │ # docs/tenants.md for why. -│ └── ns60/ # the gateway's VIP backends, two per site on one node, -│ # each with a fixed address and a VPCAttachment, -│ # answering TCP and UDP on port 80 +│ ├── ns60/ # the gateway's VIP backends, two per site on one node, +│ │ # each with a fixed address and a VPCAttachment, +│ │ # answering TCP and UDP on port 80 +│ └── policies/ # the traffic policy sets tenants bind; each +│ # tenant's own policy/ binds every attachment ├── node_files/ │ ├── dfw/ config.yaml │ ├── iad/ config.yaml @@ -450,7 +453,10 @@ deploy/containerlab/ ├── deploy-cni.sh ├── deploy-fabric.sh ├── deploy-galactic-router.sh - └── deploy-ns.sh + ├── deploy-traffic-policy.sh + ├── deploy-tenant-policy.sh + ├── deploy-ns.sh + └── publish-attachments.sh ``` ## Prerequisites @@ -458,6 +464,7 @@ deploy/containerlab/ - ContainerLab >= 0.54 - Docker - `kind` CLI +- `jq` - Host kernel with SRv6 support ## Quick start @@ -477,64 +484,83 @@ task deploy ## Tasks -| Task | Description | -| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | -| `build` | Build all container images (node, galactic-router, galactic-cni, frr, host) | -| `build:node` | Build the custom `kindest/node:galactic` image | -| `build:galactic-router` | Build the galactic-router container from Go source | -| `build:galactic-cni` | Build the galactic-cni installer image | -| `build:galactic-gateway` | Build the galactic-gateway edge load balancer image | -| `build:frr` | Build the FRR container from Alpine edge | -| `build:remote-host` | Build the off-fabric nginx host image | -| `deploy` | Build images, apply host sysctls, and deploy the lab | -| `deploy:topology` | Deploy the ContainerLab topology (transit routers) | -| `deploy:clusters` | Create the three Kind clusters and export their kubeconfigs | -| `deploy:images` | Load container images into Kind clusters | -| `deploy:system` | Install BGP and VPC CRDs; apply the galactic-system namespace and shared RBAC¹ | -| `deploy:cni` | Install Cilium and Multus, then the galactic-cni DaemonSet | -| `deploy:fabric` | Apply FRR DaemonSets to all clusters | -| `deploy:bmp-collector` | Install the BMP collector on iad-worker3, where the route reflector streams to it | -| `deploy:galactic-router` | Apply galactic-router DaemonSets and BGP CRs | -| `deploy:galactic-gateway` | Install galactic-gateway on every edge node, with each site's VIP rules (after `deploy:scenarios`) | -| `deploy:scenarios` | Deploy all VPC test scenarios | -| `deploy:ns10` | Deploy ns10 test VPC (IPv6-only, fd20 ULA) | -| `deploy:ns20` | Deploy ns20 test VPC (dual-stack, fd20 ULA + IPv4) | -| `deploy:ns30` | Deploy ns30 test VPC (dfw only, 2 pods) | -| `deploy:ns40` | Deploy ns40 test VPC (iad only, 2 pods) | -| `deploy:ns60` | Deploy ns60 test VPC (two VIP backends per site, on one node) | -| `verify:fabric-metrics` | Scrape every fabric-router's frr-exporter; fail on a failed collector or a session not Established | -| `verify:bmp` | Check the route reflector and every fabric-router stream to the BMP collector, and that it agrees with each on every session's state | -| `verify:underlay` | Ping every underlay loopback from tr1 over both IPv4 and IPv6 | -| `verify:gateway` | Check every NetworkGateway is Ready, every NetworkRule Accepted, every generated ServiceVIPBinding Bound | -| `verify:gateway-ingress` | TCP and UDP from the off-fabric host to each site's VIP, through every gateway node in turn | -| `verify:gateway-restart` | Restart dfw-worker2's gateway and shard in turn while the other carries traffic (#710) | -| `verify:gateway-detach` | Turn dfw-worker2's gateway datapath off: the ingress check must fail, then pass (not in `verify`) | -| `verify:nat-datapath` | Full egress round trip to the off-fabric host, IPv6 (NAT66) and IPv4 (NAT64) | -| `verify:nat-return-route` | Prove a reply from outside the fabric reaches each shard's masquerade addresses | -| `verify:nat-local` | Prove each site's tenants egress through that site's own edge shard | -| `verify:nat-collision` | Prove two tenants sharing a ULA on one node do not share a translation | -| `verify:mss-clamp` | Prove full-size TCP segments cross the fabric, out through NAT and across sites | -| `verify:pmtu` | Prove a packet too big for the fabric draws an ICMP error from its own node, between sites and through NAT | -| `verify:nat-icmp` | Prove ping, traceroute and path MTU discovery work through each site's egress shard | -| `verify:nat-active-active` | Hashed egress on dfw: both shards carry tenants, a failure and a drain move only theirs | -| `verify:scenarios` | Verify ping across all VPC test scenarios | -| `verify:ns10` | Verify ns10 ping (IPv6-only, 3-site mesh) | -| `verify:ns20` | Verify ns20 ping (dual-stack, 3-site mesh) | -| `verify:ns30` | Verify ns30 ping (dfw only, 2 pods) | -| `verify:ns40` | Verify ns40 ping (iad only, 2 pods) | -| `destroy` | Destroy the lab and remove all Kind clusters | -| `restart` | Full rebuild — destroy then redeploy | -| `rebuild` | Full rebuild — clean (destroy + delete images/artifacts) then redeploy | -| `inspect` | Show running nodes and management addresses | -| `graph` | Generate a draw.io diagram for the topology | -| `host-setup` | Apply required host sysctls (IPv6 forwarding, inotify limits) | -| `clean` | Destroy lab, delete built images, and remove lab artifacts | -| `test` | Run all verification checks | +| Task | Description | +| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | +| `build` | Build all container images (node, galactic-router, galactic-cni, frr, host) | +| `build:node` | Build the custom `kindest/node:galactic` image | +| `build:galactic-router` | Build the galactic-router container from Go source | +| `build:galactic-cni` | Build the galactic-cni installer image | +| `build:galactic-gateway` | Build the galactic-gateway edge load balancer image | +| `build:frr` | Build the FRR container from Alpine edge | +| `build:remote-host` | Build the off-fabric nginx host image | +| `build:galactic-controller` | Build the galactic-controller image (traffic policy's cell role) | +| `deploy` | Build images, apply host sysctls, and deploy the lab | +| `deploy:topology` | Deploy the ContainerLab topology (transit routers) | +| `deploy:clusters` | Create the three Kind clusters and export their kubeconfigs | +| `deploy:images` | Load container images into Kind clusters | +| `deploy:system` | Install BGP and VPC CRDs; apply the galactic-system namespace and shared RBAC¹ | +| `deploy:cni` | Install Cilium and Multus, then the galactic-cni DaemonSet | +| `deploy:traffic-policy` | Install the traffic policy CRDs and galactic-controller, and wait for its webhook (after `deploy:cert-manager`) | +| `deploy:fabric` | Apply FRR DaemonSets to all clusters | +| `deploy:bmp-collector` | Install the BMP collector on iad-worker3, where the route reflector streams to it | +| `deploy:galactic-router` | Apply galactic-router DaemonSets and BGP CRs | +| `deploy:galactic-gateway` | Install galactic-gateway on every edge node, with each site's VIP rules (after `deploy:scenarios`) | +| `deploy:scenarios` | Deploy all VPC test scenarios | +| `deploy:ns10` | Deploy ns10 test VPC (IPv6-only, fd20 ULA) | +| `deploy:ns20` | Deploy ns20 test VPC (dual-stack, fd20 ULA + IPv4) | +| `deploy:ns30` | Deploy ns30 test VPC (dfw only, 2 pods) | +| `deploy:ns40` | Deploy ns40 test VPC (iad only, 2 pods) | +| `deploy:ns60` | Deploy ns60 test VPC (two VIP backends per site, on one node) | +| `verify:fabric-metrics` | Scrape every fabric-router's frr-exporter; fail on a failed collector or a session not Established | +| `verify:bmp` | Check the route reflector and every fabric-router stream to the BMP collector, and that it agrees with each on every session's state | +| `verify:underlay` | Ping every underlay loopback from tr1 over both IPv4 and IPv6 | +| `verify:gateway` | Check every NetworkGateway is Ready, every NetworkRule Accepted, every generated ServiceVIPBinding Bound | +| `verify:gateway-ingress` | TCP and UDP from the off-fabric host to each site's VIP, through every gateway node in turn | +| `verify:gateway-restart` | Restart dfw-worker2's gateway and shard in turn while the other carries traffic (#710) | +| `verify:gateway-detach` | Turn dfw-worker2's gateway datapath off: the ingress check must fail, then pass (not in `verify`) | +| `verify:nat-datapath` | Full egress round trip to the off-fabric host, IPv6 (NAT66) and IPv4 (NAT64) | +| `verify:nat-return-route` | Prove a reply from outside the fabric reaches each shard's masquerade addresses | +| `verify:nat-local` | Prove each site's tenants egress through that site's own edge shard | +| `verify:nat-collision` | Prove two tenants sharing a ULA on one node do not share a translation | +| `verify:mss-clamp` | Prove full-size TCP segments cross the fabric, out through NAT and across sites | +| `verify:pmtu` | Prove a packet too big for the fabric draws an ICMP error from its own node, between sites and through NAT | +| `verify:nat-icmp` | Prove ping, traceroute and path MTU discovery work through each site's egress shard | +| `verify:nat-active-active` | Hashed egress on dfw: both shards carry tenants, a failure and a drain move only theirs | +| `verify:scenarios` | Verify ping across all VPC test scenarios | +| `verify:ns10` | Verify ns10 ping (IPv6-only, 3-site mesh) | +| `verify:ns20` | Verify ns20 ping (dual-stack, 3-site mesh) | +| `verify:ns30` | Verify ns30 ping (dfw only, 2 pods) | +| `verify:ns40` | Verify ns40 ping (iad only, 2 pods) | +| `verify:traffic-policy` | Every binding installed; each path forwards what it grants and drops, and counts, what it does not | +| `destroy` | Destroy the lab and remove all Kind clusters | +| `restart` | Full rebuild — destroy then redeploy | +| `rebuild` | Full rebuild — clean (destroy + delete images/artifacts) then redeploy | +| `inspect` | Show running nodes and management addresses | +| `graph` | Generate a draw.io diagram for the topology | +| `host-setup` | Apply required host sysctls (IPv6 forwarding, inotify limits) | +| `clean` | Destroy lab, delete built images, and remove lab artifacts | +| `test` | Run all verification checks | ¹ Network CRDs come from the latest commit on `datum-cloud/network`'s `main`, not the version `go.mod` requires. Set `NETWORK_REF=` to track a different branch, or `NETWORK_SHA=` to pin one commit. +## Traffic policy + +The lab enforces interface traffic policy (`GALACTIC_CNI_TRAFFIC_POLICY=enforce` +in `resources/galactic-cni/shared/`), while production defaults to `off`. A +tenant interface therefore forwards nothing until galactic-router publishes +its bindings, so each `deploy:nsNN` task applies the tenant's policies and +bindings (`scripts/deploy-tenant-policy.sh`, from +`resources/tenants//policy/`), deploys its pods, then publishes every +attachment's status, including the host interface the bindings attach to +(`scripts/publish-attachments.sh`, which needs `jq` on the host). The policy +sets in `resources/tenants/policies/` grant each tenant only the paths its +checks use, so a new check that sends something new needs a grant too. +`task verify:traffic-policy` proves a grant and a denial on the Private, +PublicNAT and PublicService paths and the built-in default. See +[docs/traffic-policy/rollout.md](../../docs/traffic-policy/rollout.md). + ## Verification See [docs/verification.md](docs/verification.md) for transit fabric, FRR, and galactic-router diff --git a/deploy/containerlab/Taskfile.yaml b/deploy/containerlab/Taskfile.yaml index 1081de51..3152aa67 100644 --- a/deploy/containerlab/Taskfile.yaml +++ b/deploy/containerlab/Taskfile.yaml @@ -23,6 +23,7 @@ tasks: - "build:galactic-cni" - "build:galactic-nat" - "build:galactic-gateway" + - "build:galactic-controller" - "build:remote-host" "build:node": @@ -72,6 +73,11 @@ tasks: cmds: - docker build --network=host --build-context network=../../../network -t galactic-gateway:latest -f ../../containers/galactic-gateway/Dockerfile ../.. + "build:galactic-controller": + desc: Build the galactic-controller image (traffic policy's cell role) + cmds: + - docker build --network=host --build-context network=../../../network -t galactic-controller:latest -f ../../containers/galactic-controller/Dockerfile ../.. + "build:remote-host": desc: Build the off-fabric remote host image (nginx) cmds: @@ -95,6 +101,10 @@ tasks: # mounts a csi-driver volume, and without the driver the pod (FRR # included) cannot start. - task: "deploy:cert-manager" + # After cert-manager, which issues its webhook's certificate, and + # before deploy:galactic-router, whose traffic policy reconciler + # watches the CRDs it installs. + - task: "deploy:traffic-policy" - task: "deploy:fabric" - task: "deploy:bmp-collector" - task: "deploy:galactic-router" @@ -263,6 +273,8 @@ tasks: load galactic-cni:latest {{.CNI_NODES}} load galactic-nat:latest {{.NAT_NODES}} load galactic-gateway:latest {{.GATEWAY_NODES}} + # The cell controller schedules on any untainted worker. + load galactic-controller:latest {{.FABRIC_NODES}} "deploy:system": desc: Install CRDs, namespace, and RBAC @@ -302,6 +314,11 @@ tasks: cmds: - ./scripts/deploy-fabric.sh + "deploy:traffic-policy": + desc: Install the traffic policy CRDs and galactic-controller in every cluster + cmds: + - ./scripts/deploy-traffic-policy.sh + "deploy:cert-manager": desc: Install cert-manager and its csi-driver (fabric-api certificates) cmds: @@ -345,33 +362,44 @@ tasks: "deploy:ns10": desc: Deploy ns10 test VPC (IPv6-only, fd20 ULA) cmds: + - ./scripts/deploy-tenant-policy.sh ns10 dfw sjc iad - ./scripts/deploy-ns.sh ns10 dfw sjc iad + - ./scripts/publish-attachments.sh ns10 dfw sjc iad "deploy:ns20": desc: Deploy ns20 test VPC (dual-stack) cmds: + - ./scripts/deploy-tenant-policy.sh ns20 dfw sjc iad - ./scripts/deploy-ns.sh ns20 dfw sjc iad + - ./scripts/publish-attachments.sh ns20 dfw sjc iad "deploy:ns30": desc: Deploy ns30 test VPC (dfw only, 2 pods) cmds: + - ./scripts/deploy-tenant-policy.sh ns30 dfw - ./scripts/deploy-ns.sh ns30 dfw + - ./scripts/publish-attachments.sh ns30 dfw "deploy:ns40": desc: Deploy ns40 test VPC (iad only, 2 pods) cmds: + - ./scripts/deploy-tenant-policy.sh ns40 iad - ./scripts/deploy-ns.sh ns40 iad + - ./scripts/publish-attachments.sh ns40 iad "deploy:ns60": desc: Deploy ns60 test VPC (two VIP backends per site, on one node) cmds: + - ./scripts/deploy-tenant-policy.sh ns60 dfw sjc iad - ./scripts/deploy-ns.sh ns60 dfw sjc iad - - ./scripts/publish-ns60-attachments.sh dfw sjc iad + - ./scripts/publish-attachments.sh ns60 dfw sjc iad "deploy:ns71": desc: Deploy ns71 test VPCs (overlapping ULA on one node, same-node collision fixture) cmds: + - ./scripts/deploy-tenant-policy.sh ns71 sjc - ./scripts/deploy-ns.sh ns71 sjc + - ./scripts/publish-attachments.sh ns71 sjc verify: desc: Run all verification checks @@ -400,6 +428,9 @@ tasks: - task: "verify:nat64-non-global" - task: "verify:nat-active-active" - task: "verify:scenarios" + # Last: it briefly rebinds and edits tenant policies, and restores + # them before it returns. + - task: "verify:traffic-policy" # First, because every check after it rides these bonds: a member that # never joined its aggregate leaves everything else passing over the @@ -1105,6 +1136,13 @@ tasks: - task: "verify:ns30" - task: "verify:ns40" + # Not idempotent while it runs: it edits ns60's dfw policy and unbinds + # ns30/private-b's ingress for a moment, and restores both on exit. + "verify:traffic-policy": + desc: Verify every tenant binding is installed and each path forwards what it grants and drops what it does not + cmds: + - ./scripts/verify-traffic-policy.sh + "verify:ns10": desc: Verify ns10 ping (IPv6-only, 3-site mesh) cmds: diff --git a/deploy/containerlab/docs/tenants.md b/deploy/containerlab/docs/tenants.md index b2262269..c57a3155 100644 --- a/deploy/containerlab/docs/tenants.md +++ b/deploy/containerlab/docs/tenants.md @@ -17,6 +17,12 @@ for the full per-binary ADD sequence). The `galactic-router` controller then advertises each pod's EVPN route to the route reflector, distributing reachability across sites. +The lab enforces interface traffic policy, so each tenant also binds every +attachment to a policy set in both directions (`resources/tenants//policy/`, +see `resources/tenants/policies/README.md`), and `deploy:nsNN` publishes each +attachment's status after its pods start (`scripts/publish-attachments.sh`). A +pod's interface forwards nothing until then. + They differ only in scope and addressing: | VPC | Sites | Address families | VRF interface | Notes | diff --git a/deploy/containerlab/resources/galactic-cni/shared/daemonset-patch.yaml b/deploy/containerlab/resources/galactic-cni/shared/daemonset-patch.yaml index 7d19cbfc..0f3701cc 100644 --- a/deploy/containerlab/resources/galactic-cni/shared/daemonset-patch.yaml +++ b/deploy/containerlab/resources/galactic-cni/shared/daemonset-patch.yaml @@ -26,6 +26,13 @@ spec: image: galactic-cni:latest imagePullPolicy: Never env: + # The lab enforces interface traffic policy, though production + # defaults to off: every tenant interface denies until + # galactic-router publishes its bindings, and each tenant's + # fixtures grant what the lab's checks send + # (resources/tenants/policies/, verify:traffic-policy). + - name: GALACTIC_CNI_TRAFFIC_POLICY + value: enforce # Fabric uplink the eBPF hook attaches to - name: GALACTIC_CNI_EBPF_INTERFACES value: bond0 diff --git a/deploy/containerlab/resources/galactic-controller/controller-lab-patch.yaml b/deploy/containerlab/resources/galactic-controller/controller-lab-patch.yaml new file mode 100644 index 00000000..a1616cc1 --- /dev/null +++ b/deploy/containerlab/resources/galactic-controller/controller-lab-patch.yaml @@ -0,0 +1,29 @@ +# The lab's control-plane node cannot reach pod addresses on the workers over +# Cilium's tunnel, so the kube-apiserver could never call a pod-network +# webhook; it reaches a worker's own address over the Kind network, as +# cert-manager's webhook does (scripts/deploy-cert-manager.sh). The webhook +# fails closed, so an unreachable one would refuse every TrafficPolicy. +# +# On the host network the ports must not collide with anything else on a +# worker: cert-manager's webhook holds 10260, and galactic-gateway serves its +# metrics on 8081, the base's health port. One replica: iad has a single +# untainted worker, where a second host-network replica could never bind. For +# the same reason a rolling update could never place its surge pod, which +# needs the ports the old one holds, so the pod is recreated instead. +apiVersion: apps/v1 +kind: Deployment +metadata: + name: galactic-controller + namespace: galactic-system +spec: + replicas: 1 + strategy: + type: Recreate + template: + spec: + hostNetwork: true + dnsPolicy: ClusterFirstWithHostNet + containers: + - name: controller + image: galactic-controller:latest + imagePullPolicy: Never diff --git a/deploy/containerlab/resources/galactic-controller/kustomization.yaml b/deploy/containerlab/resources/galactic-controller/kustomization.yaml new file mode 100644 index 00000000..20e8d774 --- /dev/null +++ b/deploy/containerlab/resources/galactic-controller/kustomization.yaml @@ -0,0 +1,42 @@ +# The lab's galactic-controller --role=cell: config/galactic-controller/base, +# copied into base/ at deploy time (scripts/deploy-traffic-policy.sh), with +# only what the lab needs to differ patched in (see +# controller-lab-patch.yaml for why). +resources: + - base +patches: + - path: controller-lab-patch.yaml + # With one replica, the base's minAvailable: 1 budget would block draining + # the worker it runs on. + - patch: |- + $patch: delete + apiVersion: policy/v1 + kind: PodDisruptionBudget + metadata: + name: galactic-controller + namespace: galactic-system + # Replaced rather than merged: a strategic merge keys ports on + # containerPort and would keep the base's alongside these. + - target: + kind: Deployment + name: galactic-controller + patch: |- + - op: replace + path: /spec/template/spec/containers/0/args + value: + - --role=cell + - --webhook-port=10261 + - --health-probe-bind-address=:10262 + - --metrics-bind-address=:10263 + - op: replace + path: /spec/template/spec/containers/0/ports + value: + - name: webhook + containerPort: 10261 + protocol: TCP + - name: health + containerPort: 10262 + protocol: TCP + - name: metrics + containerPort: 10263 + protocol: TCP diff --git a/deploy/containerlab/resources/galactic-router/base/router-lab-patch.yaml b/deploy/containerlab/resources/galactic-router/base/router-lab-patch.yaml index 8675980a..ca97424b 100644 --- a/deploy/containerlab/resources/galactic-router/base/router-lab-patch.yaml +++ b/deploy/containerlab/resources/galactic-router/base/router-lab-patch.yaml @@ -9,3 +9,11 @@ spec: - name: galactic-router image: galactic-router:latest imagePullPolicy: Never + env: + # The lab enforces interface traffic policy + # (resources/galactic-cni/shared/), so every router publishes + # its node's bindings. deploy:traffic-policy installs the CRDs + # this watches first. Only this plain role: the reflector has + # no tenant interfaces and refuses it. + - name: GALACTIC_ROUTER_TRAFFIC_POLICY + value: "true" diff --git a/deploy/containerlab/resources/tenants/ns10/policy/bindings.yaml b/deploy/containerlab/resources/tenants/ns10/policy/bindings.yaml new file mode 100644 index 00000000..75eb70c6 --- /dev/null +++ b/deploy/containerlab/resources/tenants/ns10/policy/bindings.yaml @@ -0,0 +1,9 @@ +apiVersion: galactic.datumapis.com/v1alpha1 +kind: TrafficPolicyBinding +metadata: + name: private +spec: + ingressPolicyRef: + name: lab-ingress + egressPolicyRef: + name: lab-egress diff --git a/deploy/containerlab/resources/tenants/ns10/policy/kustomization.yaml b/deploy/containerlab/resources/tenants/ns10/policy/kustomization.yaml new file mode 100644 index 00000000..cf4d7eaf --- /dev/null +++ b/deploy/containerlab/resources/tenants/ns10/policy/kustomization.yaml @@ -0,0 +1,7 @@ +# ns10's traffic policy: the private-nat set (../../policies/README.md), bound to +# every attachment ns10 has, in both directions. A binding is named after +# its VPCAttachment, which scripts/publish-attachments.sh creates. +namespace: ns10 +resources: + - ../../policies/private-nat + - bindings.yaml diff --git a/deploy/containerlab/resources/tenants/ns20/policy/bindings.yaml b/deploy/containerlab/resources/tenants/ns20/policy/bindings.yaml new file mode 100644 index 00000000..75eb70c6 --- /dev/null +++ b/deploy/containerlab/resources/tenants/ns20/policy/bindings.yaml @@ -0,0 +1,9 @@ +apiVersion: galactic.datumapis.com/v1alpha1 +kind: TrafficPolicyBinding +metadata: + name: private +spec: + ingressPolicyRef: + name: lab-ingress + egressPolicyRef: + name: lab-egress diff --git a/deploy/containerlab/resources/tenants/ns20/policy/kustomization.yaml b/deploy/containerlab/resources/tenants/ns20/policy/kustomization.yaml new file mode 100644 index 00000000..c37ff9c4 --- /dev/null +++ b/deploy/containerlab/resources/tenants/ns20/policy/kustomization.yaml @@ -0,0 +1,7 @@ +# ns20's traffic policy: the private-nat set (../../policies/README.md), bound to +# every attachment ns20 has, in both directions. A binding is named after +# its VPCAttachment, which scripts/publish-attachments.sh creates. +namespace: ns20 +resources: + - ../../policies/private-nat + - bindings.yaml diff --git a/deploy/containerlab/resources/tenants/ns30/policy/bindings.yaml b/deploy/containerlab/resources/tenants/ns30/policy/bindings.yaml new file mode 100644 index 00000000..9d8f8e9b --- /dev/null +++ b/deploy/containerlab/resources/tenants/ns30/policy/bindings.yaml @@ -0,0 +1,19 @@ +apiVersion: galactic.datumapis.com/v1alpha1 +kind: TrafficPolicyBinding +metadata: + name: private +spec: + ingressPolicyRef: + name: lab-ingress + egressPolicyRef: + name: lab-egress +--- +apiVersion: galactic.datumapis.com/v1alpha1 +kind: TrafficPolicyBinding +metadata: + name: private-b +spec: + ingressPolicyRef: + name: lab-ingress + egressPolicyRef: + name: lab-egress diff --git a/deploy/containerlab/resources/tenants/ns30/policy/kustomization.yaml b/deploy/containerlab/resources/tenants/ns30/policy/kustomization.yaml new file mode 100644 index 00000000..8611e374 --- /dev/null +++ b/deploy/containerlab/resources/tenants/ns30/policy/kustomization.yaml @@ -0,0 +1,7 @@ +# ns30's traffic policy: the private-nat set (../../policies/README.md), bound to +# every attachment ns30 has, in both directions. A binding is named after +# its VPCAttachment, which scripts/publish-attachments.sh creates. +namespace: ns30 +resources: + - ../../policies/private-nat + - bindings.yaml diff --git a/deploy/containerlab/resources/tenants/ns40/policy/bindings.yaml b/deploy/containerlab/resources/tenants/ns40/policy/bindings.yaml new file mode 100644 index 00000000..9d8f8e9b --- /dev/null +++ b/deploy/containerlab/resources/tenants/ns40/policy/bindings.yaml @@ -0,0 +1,19 @@ +apiVersion: galactic.datumapis.com/v1alpha1 +kind: TrafficPolicyBinding +metadata: + name: private +spec: + ingressPolicyRef: + name: lab-ingress + egressPolicyRef: + name: lab-egress +--- +apiVersion: galactic.datumapis.com/v1alpha1 +kind: TrafficPolicyBinding +metadata: + name: private-b +spec: + ingressPolicyRef: + name: lab-ingress + egressPolicyRef: + name: lab-egress diff --git a/deploy/containerlab/resources/tenants/ns40/policy/kustomization.yaml b/deploy/containerlab/resources/tenants/ns40/policy/kustomization.yaml new file mode 100644 index 00000000..2bf048c4 --- /dev/null +++ b/deploy/containerlab/resources/tenants/ns40/policy/kustomization.yaml @@ -0,0 +1,7 @@ +# ns40's traffic policy: the private set (../../policies/README.md), bound to +# every attachment ns40 has, in both directions. A binding is named after +# its VPCAttachment, which scripts/publish-attachments.sh creates. +namespace: ns40 +resources: + - ../../policies/private + - bindings.yaml diff --git a/deploy/containerlab/resources/tenants/ns60/dfw/vpcattachments.yaml b/deploy/containerlab/resources/tenants/ns60/dfw/vpcattachments.yaml index 16088656..6e308ef2 100644 --- a/deploy/containerlab/resources/tenants/ns60/dfw/vpcattachments.yaml +++ b/deploy/containerlab/resources/tenants/ns60/dfw/vpcattachments.yaml @@ -3,7 +3,7 @@ # In production the attachment controller writes these and their status. The # lab has no such controller, so deploy:ns60 applies them here and then # writes each one's status from where its pod actually runs -# (scripts/publish-ns60-attachments.sh). +# (scripts/publish-attachments.sh). apiVersion: cloud.datumapis.com/v1alpha1 kind: VPCAttachment metadata: diff --git a/deploy/containerlab/resources/tenants/ns60/iad/vpcattachments.yaml b/deploy/containerlab/resources/tenants/ns60/iad/vpcattachments.yaml index 177906a5..a4764999 100644 --- a/deploy/containerlab/resources/tenants/ns60/iad/vpcattachments.yaml +++ b/deploy/containerlab/resources/tenants/ns60/iad/vpcattachments.yaml @@ -3,7 +3,7 @@ # In production the attachment controller writes these and their status. The # lab has no such controller, so deploy:ns60 applies them here and then # writes each one's status from where its pod actually runs -# (scripts/publish-ns60-attachments.sh). +# (scripts/publish-attachments.sh). apiVersion: cloud.datumapis.com/v1alpha1 kind: VPCAttachment metadata: diff --git a/deploy/containerlab/resources/tenants/ns60/policy/bindings.yaml b/deploy/containerlab/resources/tenants/ns60/policy/bindings.yaml new file mode 100644 index 00000000..b4ae20c3 --- /dev/null +++ b/deploy/containerlab/resources/tenants/ns60/policy/bindings.yaml @@ -0,0 +1,19 @@ +apiVersion: galactic.datumapis.com/v1alpha1 +kind: TrafficPolicyBinding +metadata: + name: backend-a +spec: + ingressPolicyRef: + name: lab-ingress + egressPolicyRef: + name: lab-egress +--- +apiVersion: galactic.datumapis.com/v1alpha1 +kind: TrafficPolicyBinding +metadata: + name: backend-b +spec: + ingressPolicyRef: + name: lab-ingress + egressPolicyRef: + name: lab-egress diff --git a/deploy/containerlab/resources/tenants/ns60/policy/kustomization.yaml b/deploy/containerlab/resources/tenants/ns60/policy/kustomization.yaml new file mode 100644 index 00000000..8bd38cfd --- /dev/null +++ b/deploy/containerlab/resources/tenants/ns60/policy/kustomization.yaml @@ -0,0 +1,7 @@ +# ns60's traffic policy: the public-service set (../../policies/README.md), bound to +# every attachment ns60 has, in both directions. A binding is named after +# its VPCAttachment, which scripts/publish-attachments.sh creates. +namespace: ns60 +resources: + - ../../policies/public-service + - bindings.yaml diff --git a/deploy/containerlab/resources/tenants/ns60/sjc/vpcattachments.yaml b/deploy/containerlab/resources/tenants/ns60/sjc/vpcattachments.yaml index 835edea5..95a13e55 100644 --- a/deploy/containerlab/resources/tenants/ns60/sjc/vpcattachments.yaml +++ b/deploy/containerlab/resources/tenants/ns60/sjc/vpcattachments.yaml @@ -3,7 +3,7 @@ # In production the attachment controller writes these and their status. The # lab has no such controller, so deploy:ns60 applies them here and then # writes each one's status from where its pod actually runs -# (scripts/publish-ns60-attachments.sh). +# (scripts/publish-attachments.sh). apiVersion: cloud.datumapis.com/v1alpha1 kind: VPCAttachment metadata: diff --git a/deploy/containerlab/resources/tenants/ns71/policy/bindings.yaml b/deploy/containerlab/resources/tenants/ns71/policy/bindings.yaml new file mode 100644 index 00000000..9d8f8e9b --- /dev/null +++ b/deploy/containerlab/resources/tenants/ns71/policy/bindings.yaml @@ -0,0 +1,19 @@ +apiVersion: galactic.datumapis.com/v1alpha1 +kind: TrafficPolicyBinding +metadata: + name: private +spec: + ingressPolicyRef: + name: lab-ingress + egressPolicyRef: + name: lab-egress +--- +apiVersion: galactic.datumapis.com/v1alpha1 +kind: TrafficPolicyBinding +metadata: + name: private-b +spec: + ingressPolicyRef: + name: lab-ingress + egressPolicyRef: + name: lab-egress diff --git a/deploy/containerlab/resources/tenants/ns71/policy/kustomization.yaml b/deploy/containerlab/resources/tenants/ns71/policy/kustomization.yaml new file mode 100644 index 00000000..f83178ac --- /dev/null +++ b/deploy/containerlab/resources/tenants/ns71/policy/kustomization.yaml @@ -0,0 +1,7 @@ +# ns71's traffic policy: the private-nat set (../../policies/README.md), bound to +# every attachment ns71 has, in both directions. A binding is named after +# its VPCAttachment, which scripts/publish-attachments.sh creates. +namespace: ns71 +resources: + - ../../policies/private-nat + - bindings.yaml diff --git a/deploy/containerlab/resources/tenants/policies/README.md b/deploy/containerlab/resources/tenants/policies/README.md new file mode 100644 index 00000000..6ee26fee --- /dev/null +++ b/deploy/containerlab/resources/tenants/policies/README.md @@ -0,0 +1,17 @@ +# Lab traffic policies + +The lab enforces interface traffic policy (`GALACTIC_CNI_TRAFFIC_POLICY=enforce` +in `resources/galactic-cni/shared/`), so a tenant interface forwards only what +its bindings grant. These are the grants, one set per kind of tenant, each an +ingress and an egress policy granting only the paths that tenant's checks use: + +| Set | Tenants | Egress grants | Ingress grants | +| ---------------- | ---------------------- | ------------------ | --------------------------------- | +| `private-nat` | ns10, ns20, ns30, ns71 | Private, PublicNAT | Private (except TCP 9), PublicNAT | +| `private` | ns40 | Private | Private (except TCP 9) | +| `public-service` | ns60 | PublicService | PublicService | + +The TCP 9 deny on Private ingress, and every path a set leaves out, are what +`verify:traffic-policy` uses to prove a denial is enforced and counted. Each +tenant's `/policy/` applies one set into its namespace and binds every +attachment the tenant has to it, in both directions. diff --git a/deploy/containerlab/resources/tenants/policies/private-nat/kustomization.yaml b/deploy/containerlab/resources/tenants/policies/private-nat/kustomization.yaml new file mode 100644 index 00000000..1bf9e66a --- /dev/null +++ b/deploy/containerlab/resources/tenants/policies/private-nat/kustomization.yaml @@ -0,0 +1,2 @@ +resources: + - policies.yaml diff --git a/deploy/containerlab/resources/tenants/policies/private-nat/policies.yaml b/deploy/containerlab/resources/tenants/policies/private-nat/policies.yaml new file mode 100644 index 00000000..5f9079f4 --- /dev/null +++ b/deploy/containerlab/resources/tenants/policies/private-nat/policies.yaml @@ -0,0 +1,47 @@ +# Tenants that talk privately across the fabric and egress through the NAT +# shards: ns10 and ns20 (cross-site pings, NAT66/NAT64 egress, MSS clamping, +# path MTU errors from the node), ns30 (two attachments on dfw's compute +# node, which verify:nat-active-active spreads across dfw's shards with the +# rest) and ns71 (the NAT collision fixture). +apiVersion: galactic.datumapis.com/v1alpha1 +kind: TrafficPolicy +metadata: + name: lab-egress +spec: + rules: + - name: allow-private + action: Allow + match: + path: Private + - name: allow-nat + action: Allow + match: + path: PublicNAT +--- +apiVersion: galactic.datumapis.com/v1alpha1 +kind: TrafficPolicy +metadata: + name: lab-ingress +spec: + rules: + # verify:traffic-policy's negative check on the Private path: a TCP + # listener on port 9 is never reached, while the same listener on another + # port is. + - name: deny-discard + action: Deny + match: + path: Private + protocol: TCP + destinationPorts: + - start: 9 + end: 9 + # Also what the node itself sends the workload, such as a Packet Too Big + # (verify:pmtu). + - name: allow-private + action: Allow + match: + path: Private + - name: allow-nat-returns + action: Allow + match: + path: PublicNAT diff --git a/deploy/containerlab/resources/tenants/policies/private/kustomization.yaml b/deploy/containerlab/resources/tenants/policies/private/kustomization.yaml new file mode 100644 index 00000000..1bf9e66a --- /dev/null +++ b/deploy/containerlab/resources/tenants/policies/private/kustomization.yaml @@ -0,0 +1,2 @@ +resources: + - policies.yaml diff --git a/deploy/containerlab/resources/tenants/policies/private/policies.yaml b/deploy/containerlab/resources/tenants/policies/private/policies.yaml new file mode 100644 index 00000000..043abd3a --- /dev/null +++ b/deploy/containerlab/resources/tenants/policies/private/policies.yaml @@ -0,0 +1,31 @@ +# Tenants that only talk privately: ns40 (two attachments on one node). +# Nothing here grants PublicNAT, so its egress through a NAT shard is denied. +apiVersion: galactic.datumapis.com/v1alpha1 +kind: TrafficPolicy +metadata: + name: lab-egress +spec: + rules: + - name: allow-private + action: Allow + match: + path: Private +--- +apiVersion: galactic.datumapis.com/v1alpha1 +kind: TrafficPolicy +metadata: + name: lab-ingress +spec: + rules: + - name: deny-discard + action: Deny + match: + path: Private + protocol: TCP + destinationPorts: + - start: 9 + end: 9 + - name: allow-private + action: Allow + match: + path: Private diff --git a/deploy/containerlab/resources/tenants/policies/public-service/kustomization.yaml b/deploy/containerlab/resources/tenants/policies/public-service/kustomization.yaml new file mode 100644 index 00000000..1bf9e66a --- /dev/null +++ b/deploy/containerlab/resources/tenants/policies/public-service/kustomization.yaml @@ -0,0 +1,2 @@ +resources: + - policies.yaml diff --git a/deploy/containerlab/resources/tenants/policies/public-service/policies.yaml b/deploy/containerlab/resources/tenants/policies/public-service/policies.yaml new file mode 100644 index 00000000..b5ea1c18 --- /dev/null +++ b/deploy/containerlab/resources/tenants/policies/public-service/policies.yaml @@ -0,0 +1,25 @@ +# ns60's VIP backends: they receive the gateways' public service traffic and +# answer it, and nothing else (verify:gateway-ingress). verify:traffic-policy +# inserts a temporary deny ahead of allow-service to prove a denial on this +# path, then restores this policy. +apiVersion: galactic.datumapis.com/v1alpha1 +kind: TrafficPolicy +metadata: + name: lab-egress +spec: + rules: + - name: allow-service-replies + action: Allow + match: + path: PublicService +--- +apiVersion: galactic.datumapis.com/v1alpha1 +kind: TrafficPolicy +metadata: + name: lab-ingress +spec: + rules: + - name: allow-service + action: Allow + match: + path: PublicService diff --git a/deploy/containerlab/scripts/deploy-tenant-policy.sh b/deploy/containerlab/scripts/deploy-tenant-policy.sh new file mode 100755 index 00000000..7c6312ef --- /dev/null +++ b/deploy/containerlab/scripts/deploy-tenant-policy.sh @@ -0,0 +1,34 @@ +#!/bin/bash +# deploy-tenant-policy.sh — apply a tenant's traffic policies and bindings +# (resources/tenants//policy/) to the given sites. +# +# The lab enforces traffic policy, so a tenant's interfaces deny everything +# until these are bound and galactic-router publishes them. Runs before the +# tenant's pods, so a pod's interface is granted as soon as its attachment is +# published (scripts/publish-attachments.sh). +# +# Usage: deploy-tenant-policy.sh [site...] +set -euo pipefail + +if [ "$#" -lt 2 ]; then + echo "usage: $(basename "$0") [site...]" >&2 + exit 1 +fi + +NS="$1" +shift + +SCRIPT_DIR=$(cd "$(dirname "$0")" && pwd) +source "${SCRIPT_DIR}/lib.sh" + +for site in "$@"; do + node=$(control_plane "${site}") + echo "Applying ${NS}'s traffic policy to ${site}..." + ensure_namespace "${node}" "${NS}" + # The policy sets each tenant's policy/ kustomization pulls in. + copy_to "${node}" tenants/policies + copy_to "${node}" "tenants/${NS}/policy" + apply_k "${node}" "/galactic/resources/tenants/${NS}/policy/" +done + +echo "Done." diff --git a/deploy/containerlab/scripts/deploy-traffic-policy.sh b/deploy/containerlab/scripts/deploy-traffic-policy.sh new file mode 100755 index 00000000..9fe893c9 --- /dev/null +++ b/deploy/containerlab/scripts/deploy-traffic-policy.sh @@ -0,0 +1,67 @@ +#!/bin/bash +# deploy-traffic-policy.sh — install interface traffic policy's cell side in +# every cluster: the galactic.datumapis.com CRDs and galactic-controller +# --role=cell (its ServiceAccount, RBAC, and the lab overlay of +# config/galactic-controller/base), then wait until its fail-closed webhook +# admits a policy. +# +# Runs after deploy:cert-manager, which issues the webhook's serving +# certificate, and before deploy:galactic-router, whose traffic policy +# reconciler (GALACTIC_ROUTER_TRAFFIC_POLICY) watches these CRDs and never +# syncs without them. Each tenant's own policies and bindings are applied +# with the tenant (scripts/deploy-tenant-policy.sh). +set -euo pipefail + +SCRIPT_DIR=$(cd "$(dirname "$0")" && pwd) +source "${SCRIPT_DIR}/lib.sh" + +CONFIG=$(cd "${SCRIPT_DIR}/../../../config" && pwd) + +# A server-side dry run of a valid policy goes through the webhook and stores +# nothing, so it succeeds only once the webhook serves. +PROBE='apiVersion: galactic.datumapis.com/v1alpha1 +kind: TrafficPolicy +metadata: + name: webhook-probe + namespace: galactic-system +spec: + rules: + - name: probe + action: Deny' + +for site in dfw sjc iad; do + node=$(control_plane "${site}") + echo "Installing traffic policy CRDs and galactic-controller on ${site}..." + docker exec "${node}" rm -rf /galactic/traffic-policy + docker exec "${node}" mkdir -p /galactic/traffic-policy + docker cp "${CONFIG}/crd" "${node}:/galactic/traffic-policy/crd" + docker cp "${CONFIG}/galactic-controller" "${node}:/galactic/traffic-policy/galactic-controller" + apply_k "${node}" /galactic/traffic-policy/crd/ + docker exec "${node}" kubectl wait --for=condition=Established --timeout=60s \ + crd/trafficpolicies.galactic.datumapis.com crd/trafficpolicybindings.galactic.datumapis.com + # ServiceAccount and RBAC only; the Deployment comes from the lab overlay. + apply_k "${node}" /galactic/traffic-policy/galactic-controller/ + + # Same nesting as deploy-cni.sh: copy_to replaces the overlay, then the + # production base lands in its base/ leaf. + copy_to "${node}" galactic-controller + docker cp "${CONFIG}/galactic-controller/base" "${node}:/galactic/resources/galactic-controller/base" + apply_k "${node}" /galactic/resources/galactic-controller/ + docker exec "${node}" kubectl -n galactic-system rollout status deployment/galactic-controller --timeout=180s + + echo "Waiting for ${site}'s webhook to admit a policy..." + for i in $(seq 1 60); do + if printf '%s\n' "${PROBE}" | docker exec -i "${node}" kubectl create --dry-run=server -f - >/dev/null 2>&1; then + echo "${site}: webhook serving" + break + fi + if [ "${i}" -eq 60 ]; then + echo "error: ${site}'s galactic-controller webhook did not admit a policy within 120s" >&2 + printf '%s\n' "${PROBE}" | docker exec -i "${node}" kubectl create --dry-run=server -f - >&2 || true + exit 1 + fi + sleep 2 + done +done + +echo "Done." diff --git a/deploy/containerlab/scripts/publish-attachments.sh b/deploy/containerlab/scripts/publish-attachments.sh new file mode 100755 index 00000000..e57d7115 --- /dev/null +++ b/deploy/containerlab/scripts/publish-attachments.sh @@ -0,0 +1,89 @@ +#!/bin/bash +# publish-attachments.sh — make a tenant's VPCAttachments and write their +# status, the job the attachment controller does in production. +# +# Every galactic NetworkAttachmentDefinition in the namespace is one +# attachment, named after the NAD. One the tenant's own manifests do not +# define (only ns60's do, for the labels its NetworkRules select by) is +# created from the NAD's vpc. Each one's status then gets what its pod +# reports once scheduled: +# +# - vpc, vpcAttachment: the NAD's identifiers; +# - node, podName: where the pod using the NAD (as its +# v1.multus-cni.io/default-network) runs; +# - hostInterface: the host side of its veth, GH +# (internal/plumbing/intf), which galactic-router's traffic policy +# reconciler binds the attachment's policies to. +# +# ns60's NetworkRules wait on node to select a backend; every tenant +# interface waits on hostInterface, since the lab enforces traffic policy and +# an interface with no published bindings denies everything. +# +# Usage: publish-attachments.sh [site...] +set -euo pipefail + +if [ "$#" -lt 2 ]; then + echo "usage: $(basename "$0") [site...]" >&2 + exit 1 +fi + +NS="$1" +shift + +SCRIPT_DIR=$(cd "$(dirname "$0")" && pwd) +source "${SCRIPT_DIR}/lib.sh" + +for site in "$@"; do + cp=$(control_plane "${site}") + for nad in $(docker exec "${cp}" kubectl get network-attachment-definitions -n "${NS}" \ + -o jsonpath='{.items[*].metadata.name}'); do + config=$(docker exec "${cp}" kubectl get network-attachment-definitions -n "${NS}" "${nad}" \ + -o jsonpath='{.spec.config}') + vpc=$(jq -r '[.plugins[] | select(.type == "galactic-veth" or .type == "galactic-tap")][0].vpc // empty' <<<"${config}") + attachment=$(jq -r '[.plugins[] | select(.type == "galactic-veth" or .type == "galactic-tap")][0].vpcattachment // empty' <<<"${config}") + if [ -z "${vpc}" ] || [ -z "${attachment}" ]; then + continue # not a galactic network + fi + + if ! docker exec "${cp}" kubectl get vpcattachment -n "${NS}" "${nad}" >/dev/null 2>&1; then + docker exec -i "${cp}" kubectl apply -f - >/dev/null <&2 + exit 1 + fi + sleep 2 + done + node=$(docker exec "${cp}" kubectl get pod -n "${NS}" "${pod}" -o jsonpath='{.spec.nodeName}') + host=$(printf 'G%09s%03sH' "${vpc}" "${attachment}" | tr ' ' '0') + + docker exec "${cp}" kubectl patch vpcattachment -n "${NS}" "${nad}" --subresource=status --type=merge \ + -p "{\"status\":{\"vpc\":\"${vpc}\",\"vpcAttachment\":\"${attachment}\",\"node\":\"${node}\",\"podName\":\"${pod}\",\"hostInterface\":\"${host}\"}}" >/dev/null + echo "${site}: ${NS}/${nad} -> ${node} (${pod}, ${host})" + done +done diff --git a/deploy/containerlab/scripts/publish-ns60-attachments.sh b/deploy/containerlab/scripts/publish-ns60-attachments.sh deleted file mode 100755 index 6a67af04..00000000 --- a/deploy/containerlab/scripts/publish-ns60-attachments.sh +++ /dev/null @@ -1,40 +0,0 @@ -#!/bin/bash -# publish-ns60-attachments.sh — write the status of ns60's VPCAttachments, -# the job the attachment controller does in production. -# -# Each site's NetworkRules select ns60's backends through these attachments -# (resources/tenants/ns60//vpcattachments.yaml). An attachment is a -# backend only once its status names its VPC and its node, so this waits for -# each attachment's pod (same "instance" label) to be scheduled and copies its -# node into the attachment's status. -# -# Usage: publish-ns60-attachments.sh [site...] -set -euo pipefail - -if [ "$#" -lt 1 ]; then - echo "usage: $(basename "$0") [site...]" >&2 - exit 1 -fi - -SCRIPT_DIR=$(cd "$(dirname "$0")" && pwd) -source "${SCRIPT_DIR}/lib.sh" - -NS=ns60 -VPC=60 - -for site in "$@"; do - cp=$(control_plane "${site}") - for attachment in $(docker exec "${cp}" kubectl get vpcattachments -n "${NS}" -l app=backend \ - -o jsonpath='{.items[*].metadata.name}'); do - instance=$(docker exec "${cp}" kubectl get vpcattachment -n "${NS}" "${attachment}" \ - -o jsonpath='{.metadata.labels.instance}') - docker exec "${cp}" kubectl wait -n "${NS}" --for=condition=PodScheduled pod \ - -l "app=backend,instance=${instance}" --timeout=120s >/dev/null - pod=$(docker exec "${cp}" kubectl get pods -n "${NS}" -l "app=backend,instance=${instance}" \ - -o jsonpath='{.items[0].metadata.name}') - node=$(docker exec "${cp}" kubectl get pod -n "${NS}" "${pod}" -o jsonpath='{.spec.nodeName}') - docker exec "${cp}" kubectl patch vpcattachment -n "${NS}" "${attachment}" --subresource=status --type=merge \ - -p "{\"status\":{\"vpc\":\"${VPC}\",\"vpcAttachment\":\"${VPC}${instance}\",\"node\":\"${node}\",\"podName\":\"${pod}\"}}" >/dev/null - echo "${site}: ${NS}/${attachment} -> ${node} (${pod})" - done -done diff --git a/deploy/containerlab/scripts/verify-gateway-ingress.sh b/deploy/containerlab/scripts/verify-gateway-ingress.sh index ac7129dd..79d6c1fd 100755 --- a/deploy/containerlab/scripts/verify-gateway-ingress.sh +++ b/deploy/containerlab/scripts/verify-gateway-ingress.sh @@ -158,6 +158,8 @@ checked=0 for site in "${SITES[@]}"; do cp=$(control_plane "${site}") vip=${VIP[${site}]} + # --node checks one gateway, so a site without it is not checked at all. + [ -z "${only}" ] || [[ " ${GATEWAYS[${site}]} " == *" ${only} "* ]] || continue backends=$(pod_names "${cp}" ns60 app=backend) if [ -z "${backends}" ]; then fail "${site}: no ns60 backend pod (run deploy:ns60)" diff --git a/deploy/containerlab/scripts/verify-nat-active-active.sh b/deploy/containerlab/scripts/verify-nat-active-active.sh index 28b3b48b..9d48a275 100755 --- a/deploy/containerlab/scripts/verify-nat-active-active.sh +++ b/deploy/containerlab/scripts/verify-nat-active-active.sh @@ -131,9 +131,12 @@ members() { echo "galactic_cni_egress_pool_members{class=\"$1\",state=\"$2\"}"; # --- tenants -------------------------------------------------------------------- -# Tenant pods on the compute node, as ns/pod. +# Tenant pods on the compute node, as ns/pod. ns60's VIP backends are left +# out: the lab enforces traffic policy, and their egress grants only the +# PublicService path (resources/tenants/policies/public-service/), so they +# cannot egress through a shard at all. mapfile -t TENANTS < <(kc get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name} {.spec.nodeName}{"\n"}{end}' | - awk -v n="${COMPUTE}" '$2 == n && $1 ~ /^ns[0-9]+\// {print $1}') + awk -v n="${COMPUTE}" '$2 == n && $1 ~ /^ns[0-9]+\// && $1 !~ /^ns60\// {print $1}') [ "${#TENANTS[@]}" -ge 3 ] || die "need at least 3 tenant pods on ${COMPUTE}, found ${#TENANTS[@]}" FRESH_POD=$(kc -n ns10 get pods -o jsonpath='{.items[0].metadata.name}') FRESH_POD="ns10/${FRESH_POD}" diff --git a/deploy/containerlab/scripts/verify-traffic-policy.sh b/deploy/containerlab/scripts/verify-traffic-policy.sh new file mode 100755 index 00000000..a34754b1 --- /dev/null +++ b/deploy/containerlab/scripts/verify-traffic-policy.sh @@ -0,0 +1,272 @@ +#!/bin/bash +# verify-traffic-policy.sh — prove the lab's interface traffic policy is +# installed, forwards what each tenant's bindings grant and drops what they do +# not, on every forwarding path, and counts each decision where it was made. +# +# 1. Every TrafficPolicyBinding reports both directions Applied on its node. +# 2. Private: an ns20 TCP connection from dfw to sjc reaches a listener on +# 8080, and never reaches the same listener on 9, which sjc's ingress +# policy denies (rule deny-discard) and counts. +# 3. PublicNAT: ns10 on dfw reaches remote-host through a NAT shard; ns60's +# backend-a on dfw, whose egress grants only PublicService, does not, and +# dfw counts the no_match denial on its egress. +# 4. PublicService: with a temporary deny-udp rule ahead of ns60's +# allow-service on dfw, the dfw VIP's UDP check fails and the rule counts +# it; with the fixture restored, it passes again. +# 5. Built-in default: unbinding ns30's private-b ingress makes it deny +# everything as default-deny-all, counted under the built-in identity; +# rebinding restores it. +# +# Steps 4 and 5 change live policy; an exit trap restores the fixtures. +set -euo pipefail + +SCRIPT_DIR=$(cd "$(dirname "$0")" && pwd) +source "${SCRIPT_DIR}/lib.sh" + +METRICS_PORT=9179 +REMOTE_HOST="2001:db8:1:40::2" +BUILTIN_UID="builtin:default-deny-all:v1" +DFW=$(control_plane dfw) +SJC=$(control_plane sjc) +rc=0 + +fail() { + echo "FAIL $*" >&2 + rc=1 +} + +# metric_sum WORKER NAME LABEL=VALUE... — the sum of every NAME sample on +# WORKER's galactic-router whose labels include each LABEL="VALUE". +metric_sum() { + local worker="$1" name="$2" + shift 2 + local lines + lines=$(docker exec "${worker}" curl -fsS "http://localhost:${METRICS_PORT}/metrics" | grep "^${name}{" || true) + for kv in "$@"; do + lines=$(grep -F "${kv%%=*}=\"${kv#*=}\"" <<<"${lines}" || true) + done + awk '{s += $NF} END {printf "%d\n", s}' <<<"${lines}" +} + +# wait_applied SITE NS BINDING — until BINDING's status shows both +# directions Applied on some node, at the binding's current generation. +wait_applied() { + local cp + cp=$(control_plane "$1") + for _ in $(seq 1 60); do + if docker exec "${cp}" kubectl get trafficpolicybinding -n "$2" "$3" -o json | jq -e ' + .metadata.generation as $g + | [.status.nodes[]? + | select(.ingress.state == "Applied" and .egress.state == "Applied" + and .ingress.observedGeneration == $g)] | length > 0' >/dev/null; then + return 0 + fi + sleep 2 + done + return 1 +} + +# wait_policy_applied SITE NS POLICY — until every binding naming POLICY +# reports its current generation installed for that direction. +wait_policy_applied() { + local cp gen + cp=$(control_plane "$1") + gen=$(docker exec "${cp}" kubectl get trafficpolicy -n "$2" "$3" -o jsonpath='{.metadata.generation}') + for _ in $(seq 1 60); do + if docker exec "${cp}" kubectl get trafficpolicybindings -n "$2" -o json | jq -e \ + --arg p "$3" --argjson g "${gen}" ' + [.items[].status.nodes[]? | (.ingress, .egress) + | select(.policy.name == $p)] + | length > 0 and all(.state == "Applied" and .policy.generation == $g)' >/dev/null; then + return 0 + fi + sleep 2 + done + return 1 +} + +# tcp_reaches CP NS POD ADDRESS PORT — whether POD's TCP connection to +# [ADDRESS]:PORT is accepted within 5 seconds. +tcp_reaches() { + docker exec "$1" kubectl exec -n "$2" "$3" -- \ + timeout 6 sh -c "echo probe | socat -T 5 -u - TCP6:[$4]:$5,connect-timeout=5" >/dev/null 2>&1 +} + +# listen CP NS POD PORT — a one-shot TCP listener on PORT in POD, for 20s. +listen() { + docker exec "$1" kubectl exec -n "$2" "$3" -- \ + timeout 20 socat -u "TCP6-LISTEN:$4,reuseaddr" OPEN:/dev/null >/dev/null 2>&1 & + sleep 1 +} + +out=$(mktemp) +# restore re-applies the two fixtures steps 4 and 5 change and waits for the +# router to install them, so a check run after this one sees the lab as +# deployed. +restore() { + set +e + apply_k "${DFW}" /galactic/resources/tenants/ns60/policy/ >/dev/null + apply_k "${DFW}" /galactic/resources/tenants/ns30/policy/ >/dev/null + wait_policy_applied dfw ns60 lab-ingress || echo "warning: ns60's lab-ingress not reinstalled on dfw" >&2 + wait_applied dfw ns30 private-b || echo "warning: ns30/private-b not rebound on dfw" >&2 + rm -f "${out}" +} +trap restore EXIT + +echo "--- 1. every binding is installed on its node" +for site in dfw sjc iad; do + cp=$(control_plane "${site}") + while read -r ns name; do + [ -z "${ns}" ] && continue + if wait_applied "${site}" "${ns}" "${name}"; then + echo "ok ${site}: ${ns}/${name} applied in both directions" + else + fail "${site}: ${ns}/${name} is not applied in both directions:" + docker exec "${cp}" kubectl get trafficpolicybinding -n "${ns}" "${name}" -o jsonpath='{.status}' >&2 + echo >&2 + fi + done < <(docker exec "${cp}" kubectl get trafficpolicybindings -A \ + -o jsonpath='{range .items[*]}{.metadata.namespace} {.metadata.name}{"\n"}{end}') +done + +echo "--- 2. Private: a granted port connects, a denied one does not" +src=$(pod_name "${DFW}" ns20) +dst=$(pod_name "${SJC}" ns20) +dst_ip=$(pod_ip6 "${SJC}" ns20 "${dst}") +listen "${SJC}" ns20 "${dst}" 8080 +if tcp_reaches "${DFW}" ns20 "${src}" "${dst_ip}" 8080; then + echo "ok ns20 dfw -> sjc TCP 8080 connects" +else + fail "ns20 dfw -> sjc TCP 8080 did not connect" +fi +before=$(metric_sum sjc-worker galactic_cni_traffic_policy_rule_hits_total \ + namespace=ns20 direction=ingress rule=deny-discard) +listen "${SJC}" ns20 "${dst}" 9 +if tcp_reaches "${DFW}" ns20 "${src}" "${dst_ip}" 9; then + fail "ns20 dfw -> sjc TCP 9 connected through a deny rule" +else + echo "ok ns20 dfw -> sjc TCP 9 is dropped" +fi +after=$(metric_sum sjc-worker galactic_cni_traffic_policy_rule_hits_total \ + namespace=ns20 direction=ingress rule=deny-discard) +if [ "${after}" -gt "${before}" ]; then + echo "ok sjc counted the denial on ns20's ingress: deny-discard ${before} -> ${after}" +else + fail "sjc's deny-discard counter for ns20 did not move (${before} -> ${after})" +fi + +echo "--- 3. PublicNAT: granted egress reaches remote-host, ungranted does not" +ns10=$(pod_name "${DFW}" ns10) +if tcp_reaches "${DFW}" ns10 "${ns10}" "${REMOTE_HOST}" 80; then + echo "ok ns10 dfw -> remote-host TCP 80 connects through a NAT shard" +else + fail "ns10 dfw -> remote-host TCP 80 did not connect" +fi +backend=$(pod_name "${DFW}" ns60 app=backend,instance=a) +before=$(metric_sum dfw-worker galactic_cni_traffic_policy_fallback_packets_total \ + namespace=ns60 attachment=backend-a direction=egress reason=no_match) +if tcp_reaches "${DFW}" ns60 "${backend}" "${REMOTE_HOST}" 80; then + fail "ns60 backend-a -> remote-host connected though its egress grants only PublicService" +else + echo "ok ns60 backend-a -> remote-host is dropped" +fi +after=$(metric_sum dfw-worker galactic_cni_traffic_policy_fallback_packets_total \ + namespace=ns60 attachment=backend-a direction=egress reason=no_match) +if [ "${after}" -gt "${before}" ]; then + echo "ok dfw counted backend-a's egress as no_match: ${before} -> ${after}" +else + fail "dfw's no_match counter for backend-a's egress did not move (${before} -> ${after})" +fi + +echo "--- 4. PublicService: a deny rule stops the VIP's UDP, and lifting it restores it" +docker exec -i "${DFW}" kubectl apply -f - >/dev/null <<'EOF' +apiVersion: galactic.datumapis.com/v1alpha1 +kind: TrafficPolicy +metadata: + name: lab-ingress + namespace: ns60 +spec: + rules: + - name: deny-udp + action: Deny + match: + path: PublicService + protocol: UDP + - name: allow-service + action: Allow + match: + path: PublicService +EOF +if ! wait_policy_applied dfw ns60 lab-ingress; then + fail "dfw did not install ns60's edited lab-ingress" +fi +before=$(metric_sum dfw-worker galactic_cni_traffic_policy_rule_hits_total \ + namespace=ns60 direction=ingress rule=deny-udp) +if "${SCRIPT_DIR}/verify-gateway-ingress.sh" --node dfw-worker2 >"${out}" 2>&1; then + fail "dfw's VIP passed its TCP and UDP check with UDP denied to its backends" +elif grep -q 'udp from port .*: no reply' "${out}" && ! grep -q 'tcp from port' "${out}"; then + echo "ok dfw's VIP still serves TCP, and its UDP draws no reply while denied" +else + fail "dfw's VIP check failed, but not only on UDP:" + sed 's/^/ /' "${out}" >&2 +fi +after=$(metric_sum dfw-worker galactic_cni_traffic_policy_rule_hits_total \ + namespace=ns60 direction=ingress rule=deny-udp) +if [ "${after}" -gt "${before}" ]; then + echo "ok dfw counted the VIP's UDP under deny-udp: ${before} -> ${after}" +else + fail "dfw's deny-udp counter for ns60 did not move (${before} -> ${after})" +fi +apply_k "${DFW}" /galactic/resources/tenants/ns60/policy/ >/dev/null +if ! wait_policy_applied dfw ns60 lab-ingress; then + fail "dfw did not reinstall ns60's lab-ingress" +fi +if "${SCRIPT_DIR}/verify-gateway-ingress.sh" --node dfw-worker2 >"${out}" 2>&1; then + echo "ok dfw's VIP serves TCP and UDP again" +else + fail "dfw's VIP still fails with the fixture restored:" + sed 's/^/ /' "${out}" >&2 +fi + +echo "--- 5. Built-in default: an unbound direction denies, rebinding restores it" +a=$(pod_name "${DFW}" ns30) +b=$(pod_name "${DFW}" ns30 app=private-b) +b_ip=$(pod_ip6 "${DFW}" ns30 "${b}") +docker exec "${DFW}" kubectl patch trafficpolicybinding -n ns30 private-b --type=json \ + -p '[{"op":"remove","path":"/spec/ingressPolicyRef"}]' >/dev/null +for _ in $(seq 1 60); do + got=$(docker exec "${DFW}" kubectl get trafficpolicybinding -n ns30 private-b \ + -o jsonpath='{.status.nodes[0].ingress.policy.uid}') + [ "${got}" = "${BUILTIN_UID}" ] && break + sleep 2 +done +if [ "${got}" != "${BUILTIN_UID}" ]; then + fail "ns30/private-b's ingress did not fall back to the built-in default (uid ${got:-none})" +fi +before=$(metric_sum dfw-worker galactic_cni_traffic_policy_rule_hits_total \ + namespace=ns30 attachment=private-b direction=ingress policy_uid="${BUILTIN_UID}") +if docker exec "${DFW}" kubectl exec -n ns30 "${a}" -- ping -6 -c 3 -W 2 "${b_ip}" >/dev/null 2>&1; then + fail "ns30 private -> private-b answered with private-b on the built-in default" +else + echo "ok ns30 private -> private-b is dropped on the built-in default" +fi +after=$(metric_sum dfw-worker galactic_cni_traffic_policy_rule_hits_total \ + namespace=ns30 attachment=private-b direction=ingress policy_uid="${BUILTIN_UID}") +if [ "${after}" -gt "${before}" ]; then + echo "ok dfw counted the default deny-all as a rule hit: ${before} -> ${after}" +else + fail "dfw's default deny-all counter for ns30/private-b did not move (${before} -> ${after})" +fi +apply_k "${DFW}" /galactic/resources/tenants/ns30/policy/ >/dev/null +if wait_applied dfw ns30 private-b && + docker exec "${DFW}" kubectl exec -n ns30 "${a}" -- ping -6 -c 3 -W 2 "${b_ip}" >/dev/null 2>&1; then + echo "ok ns30 private -> private-b answers again once rebound" +else + fail "ns30 private -> private-b does not answer after rebinding" +fi + +if [ "${rc}" -ne 0 ]; then + echo "traffic policy: FAILED" >&2 + exit 1 +fi +echo "traffic policy: every path forwards what it grants and drops, and counts, what it does not" diff --git a/docs/README.md b/docs/README.md index fed97eba..8383dfea 100644 --- a/docs/README.md +++ b/docs/README.md @@ -60,6 +60,9 @@ data flow, module reference, known constraints): - [traffic-policy/controller.md](traffic-policy/controller.md) — `galactic-controller --role=cell`: the TrafficPolicy webhook, the reserved default policy, binding resolution status, flags and install order. +- [traffic-policy/rollout.md](traffic-policy/rollout.md) — turning traffic + policy on in a cell: install order, writing grants, auditing with the + counters before enforcing, and backing out. - [traffic-policy/node.md](traffic-policy/node.md) — galactic-router's node side: enabling it, what each binding applies, per-node status, the `galactic_cni_traffic_policy_*` metrics and the `policy explain` and diff --git a/docs/agents/ARCHITECTURE-GATEWAY.md b/docs/agents/ARCHITECTURE-GATEWAY.md index 5d6bfc38..f6820d7e 100644 --- a/docs/agents/ARCHITECTURE-GATEWAY.md +++ b/docs/agents/ARCHITECTURE-GATEWAY.md @@ -904,7 +904,7 @@ that tag into `config/galactic-gateway/base`. - **`XDP_REDIRECT` into a veth needs NAPI enabled on the *peer*, which is a lab-only concern.** `edgedsr.c` returns `XDP_REDIRECT` whenever the route to the backend egresses an interface other than the ingress one. In native mode that calls the egress device's `ndo_xdp_xmit`, and veth's implementation silently discards the frame unless the peer end has NAPI enabled — which for a veth means the peer runs its own XDP program or has GRO turned on. A containerlab peer inside an FRR/transit container has neither by default, so a redirect there can fail exactly the way this datapath's original bug did: no drop counter moves, because `bpf_redirect()` itself succeeded and the discard happens later in `xdp_do_redirect`. Both programs are affected, and `edge_return` more so: it attaches to compute-facing links, which are veth pairs in the lab and may be veth or a plain NIC in production. `bpftool prog tracelog` and the `xdp:xdp_redirect_err` tracepoint are the diagnostics; `ethtool -K gro on` is the lab fix. A production uplink (physical NIC, or a bond slave per the bullet above) implements `ndo_xdp_xmit` natively and has no peer to ask about. This is the redirect-side analogue of the `XDP_TX`-on-veth observability quirk noted above. - **The return path is opt-in per node and fails closed at startup, not silently.** `edge_return` attaches only where `GALACTIC_GATEWAY_INTERNAL_INTERFACES` names an interface; a node that needs it and does not set it drops every reply in `KUBE-FORWARD` with nothing to say why, exactly as before this program existed. Where it *is* set, an attach failure is fatal to `setupGatewayDatapath` and the pod crash-loops rather than running with the forward half working and replies dying — the same all-or-nothing choice `edgeattach.Attach` already makes across a bond's slaves. - **A backend in another site replies through that site's edge node.** The gateway can pick a backend anywhere its `NetworkRule` reaches, and the reply then leaves through the edge node in front of *that* backend, whose own `vip_addr_table` must hold the same anycast VIP or the reply meets the unmodified kernel path and dies. Within one cluster the reconciler guarantees this; across clusters it depends on the same `NetworkRule` existing on both sides, which nothing in this repo enforces. -- **Backends depend on an attachment controller outside this repo.** `selectRuleBackends` skips a `VPCAttachment` whose `status.vpc` is not the rule's VPC, and keeps a selected one pending until its `status.node` is set. Galactic writes neither field; controllers in datum-cloud/cloud do, `status.node` from the attachment's `BGPAdvertisement`. Without them an attachment with no `status.vpc` is silently unselected, one with `status.vpc` but no `status.node` is pending with `status.node not set by datum-cloud/cloud`, and a rule with no other backend never loads; the containerlab lab patches the status by hand (`deploy/containerlab/scripts/publish-ns60-attachments.sh`). See #821. +- **Backends depend on an attachment controller outside this repo.** `selectRuleBackends` skips a `VPCAttachment` whose `status.vpc` is not the rule's VPC, and keeps a selected one pending until its `status.node` is set. Galactic writes neither field; controllers in datum-cloud/cloud do, `status.node` from the attachment's `BGPAdvertisement`. Without them an attachment with no `status.vpc` is silently unselected, one with `status.vpc` but no `status.node` is pending with `status.node not set by datum-cloud/cloud`, and a rule with no other backend never loads; the containerlab lab patches the status by hand (`deploy/containerlab/scripts/publish-attachments.sh`). See #821. - **IPv4 VIPs are accepted but not served.** A rule that predates the CRD's IPv6-only rule can still hold an IPv4 VIP, but `edge_lb` and `edge_return` parse only IPv6, `edgemap` refuses IPv4 keys, and backend nodes translate only IPv6 VIPs. So `buildDesiredRule` leaves IPv4 VIPs, plain or IPv4-mapped, out of the engine, no `-v4` advertisement is made, and each gateway node reports `/IPv4Served=False` (`Unsupported`) on the rule. The address never draws traffic that nothing answers, and the rule's IPv6 VIP keeps serving. #705 tracks serving IPv4 VIPs. - **`vip_table` has no active GC beyond crash-recovery reconcile.** By design (see Key Design Decisions above) — DSR keeps no flow state to leak in the first place. - **Egress is out of this binary's scope, not unimplemented.** Egress translation is a separate, sharded tier (`galactic-nat`, its own binary — see `cmd/galactic-nat` and `internal/controller/egressshard_controller.go`). `NetworkRule`/this datapath remain ingress-only: external client → VIP → tenant backend. diff --git a/docs/enhancements/networking/interface-traffic-policy/README.md b/docs/enhancements/networking/interface-traffic-policy/README.md index 873d8c35..e18df01a 100644 --- a/docs/enhancements/networking/interface-traffic-policy/README.md +++ b/docs/enhancements/networking/interface-traffic-policy/README.md @@ -777,7 +777,9 @@ enforcement to 6.8, which the pull requests call out. Default deny is a deliberate behavior change. The enforcement mode in `tp_state` is `off`, `audit` or `enforce`, set per node by -`GALACTIC_CNI_TRAFFIC_POLICY` and defaulting to `off` until the final change. +`GALACTIC_CNI_TRAFFIC_POLICY`. It defaults to `off`; only the containerlab lab +enforces by default. [docs/traffic-policy/rollout.md](../../../traffic-policy/rollout.md) +is the operator's step-by-step guide. - `off` changes nothing. Maps and programs are loaded, but `tp_capture` and `tp_decide` return immediately. diff --git a/docs/gateway/configuration.md b/docs/gateway/configuration.md index ef3f5f4c..f1a4b579 100644 --- a/docs/gateway/configuration.md +++ b/docs/gateway/configuration.md @@ -522,7 +522,7 @@ attachment whose `status.vpc` is set but whose `status.node` is not stays pending with `status.node not set by datum-cloud/cloud`. A rule whose attachments are all pending or unselected does not load. The containerlab lab runs none of these controllers, so -`deploy/containerlab/scripts/publish-ns60-attachments.sh` patches +`deploy/containerlab/scripts/publish-attachments.sh` patches `status.vpc` and `status.node` in by hand. Every accepted, non-deleting `NetworkRule` in the namespace is served by diff --git a/docs/traffic-policy/rollout.md b/docs/traffic-policy/rollout.md new file mode 100644 index 00000000..1567f307 --- /dev/null +++ b/docs/traffic-policy/rollout.md @@ -0,0 +1,136 @@ +# Rolling out interface traffic policy + +Interface traffic policy denies by default: once a node enforces it, a tenant +interface forwards only what its bindings grant, in each direction. That is a +deliberate behavior change, so turning it on in a cell is a migration. This +guide is the order to do it in, what to check at each step, and how to back +out. The design is in +[the traffic policy proposal](../enhancements/networking/interface-traffic-policy/README.md); +[controller.md](controller.md) and [node.md](node.md) are the component +references. + +TrafficPolicy is not Kubernetes NetworkPolicy and is unrelated to BGPPolicy: +it filters a VPC workload's own interface, by forwarding path, address, +protocol and port, and neither reads nor replaces either of those. + +## The three modes + +`GALACTIC_CNI_TRAFFIC_POLICY` on `galactic-cni` sets each node's mode: + +| Mode | Decides | Counts | Drops | Use | +| --------- | ------- | ------ | ----- | ------------------------------------------ | +| `off` | no | no | no | The default; the programs are loaded inert | +| `audit` | yes | yes | no | Measuring what enforcement would deny | +| `enforce` | yes | yes | yes | The steady state | + +`audit` is for measuring a migration, not a compatibility mode: it forwards +everything, so a cell left in it enforces nothing. There is no legacy allow, +and nothing generates reverse rules: a workload that answers connections needs +an ingress grant for the requests and an egress grant for the replies. + +In `audit` and `enforce`, a new interface's host side stays down until every +enforcement program is attached and verified, and CNI ADD fails on a node +whose kernel failed the preflight (Linux 6.8 or later, `HASH_OF_MAPS` and +`bpf_for_each_map_elem`). Production and staging run Talos v1.13.10, on +kernel 6.18. + +## Before you start + +Check every node of the cell: + +- runs a galactic release that includes traffic policy, for `galactic-cni`, + `galactic-router` and `galactic-controller`; +- has the `galactic.datumapis.com` CRDs (`config/crd/`), cert-manager, and the + cloud operator's VPCAttachment CRD installed; +- publishes `status.node` and `status.hostInterface` on every VPCAttachment, + which the cloud operator writes. An attachment without them is never bound + on any node, and denies everything once enforced. + +## Steps + +1. **Install the cell controller.** Apply `config/galactic-controller/` (its + ServiceAccount and RBAC), then `config/galactic-controller/base/`. Its + webhook fails closed, so wait until it admits a policy before anyone + writes one: a server-side dry run of a valid TrafficPolicy succeeds. It + provisions `default-deny-all` in every namespace that holds an attachment. + +2. **Turn on the node reconciler.** Set `GALACTIC_ROUTER_TRAFFIC_POLICY=true` + on the plain `galactic-router` DaemonSet, never the route reflector. With + the mode still `off`, nothing changes for any packet, but every binding + now reports what each node would install in `status.nodes`, and + `galactic_cni_traffic_policy_*` appears on each router's metrics port. + +3. **Write the grants.** For every VPCAttachment, create a + TrafficPolicyBinding of the same name with an ingress and an egress + reference, and the policies they name. A direction without a reference + gets the built-in `default-deny-all`, which denies everything. Grant by + path (`Private`, `PublicNAT`, `PublicService`) as well as by address and + port, so a rule for one path never opens another. Check that every + binding's `status.nodes` shows both directions `Applied`: + + ```sh + kubectl get trafficpolicybindings -A -o json | jq -r ' + .items[] | . as $b | .status.nodes[]? + | select(.ingress.state != "Applied" or .egress.state != "Applied") + | "\($b.metadata.namespace)/\($b.metadata.name) on \(.node): \(.ingress.state)/\(.egress.state)"' + ``` + +4. **Recreate workloads that predate the release.** An interface created + before the release gets its policy state from `galactic-cni` at startup, + but not its addresses, so its ARP and its neighbor discovery from a + global address count as ordinary traffic rather than link operations, and + are denied unless a rule grants them. Recreate those workloads; a new + interface records its addresses at ADD. + +5. **Audit.** Set `GALACTIC_CNI_TRAFFIC_POLICY=audit` on `galactic-cni`, one + node or one site at a time. Every decision is now counted, and nothing is + dropped. Leave it long enough to see each workload's normal traffic, + including periodic jobs, then find what enforcement would deny: + + ```promql + # Would be denied by no matching rule, or by a missing binding. + sum by (namespace, attachment, direction, reason) ( + increase(galactic_cni_traffic_policy_fallback_packets_total[1h]) + ) > 0 + + # Would be denied by a rule, including the built-in default. + sum by (namespace, attachment, direction, policy, rule) ( + increase(galactic_cni_traffic_policy_rule_hits_total{action="deny"}[1h]) + ) > 0 + + # Interfaces still on the built-in default. + count by (namespace, attachment, direction) ( + galactic_cni_traffic_policy_rule_hits_total{policy_uid="builtin:default-deny-all:v1"} + ) + ``` + + `policy_unavailable` means an interface has no installed policy at all: + its attachment is not bound on its node yet, or its binding's reference is + missing. `classification_error` means a packet's path could not be + determined, typically because the node's NAT shard locators have not + synced. Fix the grants and repeat until each query is empty, or shows only + traffic you mean to deny. `galactic-router policy explain` decides a + packet against a policy offline, to check a rule before relying on it. + +6. **Enforce.** Set `GALACTIC_CNI_TRAFFIC_POLICY=enforce`, again a node or a + site at a time, and watch the same queries: they now count drops. + Changing the mode takes effect at once on every existing interface and + needs no workload restart. + +## Backing out + +Set `GALACTIC_CNI_TRAFFIC_POLICY=off`. The datapath stops deciding at once and +forwards as before; bindings, status and counters stay, so moving forward +again resumes where it stopped. Turning the reconciler off +(`GALACTIC_ROUTER_TRAFFIC_POLICY=false`) leaves the installed snapshots in +place but stops updating them, so do that only with the mode `off`. + +## In the containerlab lab + +The lab in `deploy/containerlab/` enforces from the start: its `galactic-cni` +sets `enforce`, its routers run the reconciler, and every tenant fixture +binds both directions to a policy set granting only the paths its checks use +(`resources/tenants/policies/`). `task verify:traffic-policy` checks that +every binding is installed, and that each path forwards what it grants and +drops, and counts, what it does not. It is the place to try a policy change +before writing it for a cell.