From 221ea1ad235a915176c912f46ff94ed7762d5329 Mon Sep 17 00:00:00 2001 From: Peter Sprygada Date: Thu, 15 Oct 2026 11:57:32 -0400 Subject: [PATCH] feat(lab): Enforce traffic policy in the containerlab lab The containerlab lab now runs interface traffic policy end to end, in enforce mode, while production stays off. It installs the galactic.datumapis.com CRDs and galactic-controller (on the host network, since the control plane cannot reach worker pod addresses, like cert-manager's webhook), turns on the router's reconciler, and binds every tenant attachment, in both directions, to a policy set granting only the paths that tenant's checks use. A generic attachment publisher replaces the ns60-only one and writes every attachment's status, including the host interface the bindings attach to. verify:traffic-policy proves each binding is installed and, for the Private, PublicNAT and PublicService paths and the built-in default, that a grant forwards and a denial drops and is counted. Two envtest-backed tests keep the fixtures valid and every tenant NAD bound. docs/traffic-policy/ rollout.md is the operator guide for turning policy on in a cell. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 2 +- api/v1alpha1/labfixtures_test.go | 225 +++++++++++++++ deploy/containerlab/AGENTS.md | 4 +- deploy/containerlab/README.md | 140 +++++---- deploy/containerlab/Taskfile.yaml | 40 ++- deploy/containerlab/docs/tenants.md | 6 + .../galactic-cni/shared/daemonset-patch.yaml | 7 + .../controller-lab-patch.yaml | 29 ++ .../galactic-controller/kustomization.yaml | 42 +++ .../base/router-lab-patch.yaml | 8 + .../tenants/ns10/policy/bindings.yaml | 9 + .../tenants/ns10/policy/kustomization.yaml | 7 + .../tenants/ns20/policy/bindings.yaml | 9 + .../tenants/ns20/policy/kustomization.yaml | 7 + .../tenants/ns30/policy/bindings.yaml | 19 ++ .../tenants/ns30/policy/kustomization.yaml | 7 + .../tenants/ns40/policy/bindings.yaml | 19 ++ .../tenants/ns40/policy/kustomization.yaml | 7 + .../tenants/ns60/dfw/vpcattachments.yaml | 2 +- .../tenants/ns60/iad/vpcattachments.yaml | 2 +- .../tenants/ns60/policy/bindings.yaml | 19 ++ .../tenants/ns60/policy/kustomization.yaml | 7 + .../tenants/ns60/sjc/vpcattachments.yaml | 2 +- .../tenants/ns71/policy/bindings.yaml | 19 ++ .../tenants/ns71/policy/kustomization.yaml | 7 + .../resources/tenants/policies/README.md | 17 ++ .../policies/private-nat/kustomization.yaml | 2 + .../policies/private-nat/policies.yaml | 47 +++ .../policies/private/kustomization.yaml | 2 + .../tenants/policies/private/policies.yaml | 31 ++ .../public-service/kustomization.yaml | 2 + .../policies/public-service/policies.yaml | 25 ++ .../scripts/deploy-tenant-policy.sh | 34 +++ .../scripts/deploy-traffic-policy.sh | 67 +++++ .../scripts/publish-attachments.sh | 89 ++++++ .../scripts/publish-ns60-attachments.sh | 40 --- .../scripts/verify-gateway-ingress.sh | 2 + .../scripts/verify-nat-active-active.sh | 7 +- .../scripts/verify-traffic-policy.sh | 272 ++++++++++++++++++ docs/README.md | 3 + docs/agents/ARCHITECTURE-GATEWAY.md | 2 +- .../interface-traffic-policy/README.md | 4 +- docs/gateway/configuration.md | 2 +- docs/traffic-policy/rollout.md | 136 +++++++++ 44 files changed, 1321 insertions(+), 108 deletions(-) create mode 100644 api/v1alpha1/labfixtures_test.go create mode 100644 deploy/containerlab/resources/galactic-controller/controller-lab-patch.yaml create mode 100644 deploy/containerlab/resources/galactic-controller/kustomization.yaml create mode 100644 deploy/containerlab/resources/tenants/ns10/policy/bindings.yaml create mode 100644 deploy/containerlab/resources/tenants/ns10/policy/kustomization.yaml create mode 100644 deploy/containerlab/resources/tenants/ns20/policy/bindings.yaml create mode 100644 deploy/containerlab/resources/tenants/ns20/policy/kustomization.yaml create mode 100644 deploy/containerlab/resources/tenants/ns30/policy/bindings.yaml create mode 100644 deploy/containerlab/resources/tenants/ns30/policy/kustomization.yaml create mode 100644 deploy/containerlab/resources/tenants/ns40/policy/bindings.yaml create mode 100644 deploy/containerlab/resources/tenants/ns40/policy/kustomization.yaml create mode 100644 deploy/containerlab/resources/tenants/ns60/policy/bindings.yaml create mode 100644 deploy/containerlab/resources/tenants/ns60/policy/kustomization.yaml create mode 100644 deploy/containerlab/resources/tenants/ns71/policy/bindings.yaml create mode 100644 deploy/containerlab/resources/tenants/ns71/policy/kustomization.yaml create mode 100644 deploy/containerlab/resources/tenants/policies/README.md create mode 100644 deploy/containerlab/resources/tenants/policies/private-nat/kustomization.yaml create mode 100644 deploy/containerlab/resources/tenants/policies/private-nat/policies.yaml create mode 100644 deploy/containerlab/resources/tenants/policies/private/kustomization.yaml create mode 100644 deploy/containerlab/resources/tenants/policies/private/policies.yaml create mode 100644 deploy/containerlab/resources/tenants/policies/public-service/kustomization.yaml create mode 100644 deploy/containerlab/resources/tenants/policies/public-service/policies.yaml create mode 100755 deploy/containerlab/scripts/deploy-tenant-policy.sh create mode 100755 deploy/containerlab/scripts/deploy-traffic-policy.sh create mode 100755 deploy/containerlab/scripts/publish-attachments.sh delete mode 100755 deploy/containerlab/scripts/publish-ns60-attachments.sh create mode 100755 deploy/containerlab/scripts/verify-traffic-policy.sh create mode 100644 docs/traffic-policy/rollout.md diff --git a/AGENTS.md b/AGENTS.md index a2577f4e..b4d24d91 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -102,7 +102,7 @@ See [docs/node-labels.md](docs/node-labels.md) for the full strategy — why `ga - **`config/fabric-api/`** — the fabric looking glass's cell-wide prerequisites and per-cell gateway. `kubectl apply -k config/fabric-api/` applies only the `fabric-api-gateway` and `fabric-api-janitor` ServiceAccounts and RBAC (the janitor's delete access is deliberately a separate identity from the gateway's, which can only read `FabricQuery` and write its status); safe and idempotent in any cell. `config/fabric-api/base/` (the 2-replica leader-elected gateway Deployment, its ClusterIP debug Service, a PodDisruptionBudget and the `*/15` janitor CronJob) is **not** included in that kustomization and is **not** applied as-is, for the same reason as `config/galactic-gateway/base/`: it needs per-cell values — a `fabric-api` ConfigMap (`site`, `cell`, `cluster-name`) and the cell name in its certificate's URI SAN — supplied by a per-cell overlay; see `deploy/containerlab/resources/fabric-api/gateway/` for a worked example. `config/fabric-api/crd/` is a generated `FabricQuery` CRD for the lab and tests only; infra owns the CRD (and the per-cell `fabric-api` cert-manager `Issuer`) in every cell. The node sidecar is the `config/fabric-router/components/fabric-api` Component, which also adds a `fabric-api-certs` DaemonSet (`fabric-api certsync`) that owns the cert-manager csi-driver volume and copies the certificate to the node's `/run/fabric-api/tls` for the sidecar: a csi volume blocks its pod until the certificate is issued, so keeping it out of the `fabric-router` pod means a broken Issuer stalls only the certs pod, never FRR. The overlay's `CELL_NAME` replacement applies to that DaemonSet's `fabric-api-csi` volume. Not part of the root `config/kustomization.yaml`: fabric-api is opt-in. See [docs/fabric-api/configuration.md](docs/fabric-api/configuration.md) and [docs/fabric-api/api.md](docs/fabric-api/api.md). - **`config/crd/`** and **`config/galactic-controller/`** — the `galactic.datumapis.com` CRDs (TrafficPolicy, TrafficPolicyBinding) and `galactic-controller --role=cell`, the TrafficPolicy validating webhook (fail closed), default-policy provisioning and binding resolution. `kubectl apply -k config/galactic-controller/` applies only its ServiceAccount and RBAC; `base/` (two-replica Deployment, webhook Service, cert-manager serving certificate, ValidatingWebhookConfiguration) is installed with the CRDs (including the cloud operator's `VPCAttachment`, which it watches) and cert-manager in place. Neither is in the root `config/kustomization.yaml`. The node side is `galactic-router` with `GALACTIC_ROUTER_TRAFFIC_POLICY=true` (off by default; `internal/trafficpolicy/node`), which publishes each attachment's bound policies into the uSID datapath, writes the binding's `status.nodes` and exports `galactic_cni_traffic_policy_*`; galactic-cni's `GALACTIC_CNI_TRAFFIC_POLICY` decides whether the datapath enforces them. See [docs/traffic-policy/controller.md](docs/traffic-policy/controller.md) and [docs/traffic-policy/node.md](docs/traffic-policy/node.md). - **`config/monitoring/`** — Prometheus Operator resources for BGP and NAT: a `fabric-router` PodMonitor (frr-exporter, port `frr-metrics`/`9342`, as `job=fabric-router`; fabric-config-agent, port `agent-metrics`/`9343`, as `job=fabric-config-agent`), a `galactic-router` PodMonitor covering both roles (port `metrics`/`9179`), a `galactic-nat` PodMonitor for the egress shards (port `metrics`/`9182`), each adding a `node` label, and the `galactic-bgp` PrometheusRule: underlay session down, administratively down for an hour, flapping, received prefixes halved, exporter collector or scrape failing, a session announcing an IPv4-mapped next hop, BGP routes not installed, the agent's metrics scrape down, a `galactic-router` BMP station or metrics scrape down, `galactic-router` routes failing to install, an egress shard's session table evicting live sessions, an egress shard out of masquerade ports, a stalled or persistently lossy NIC receive queue on a `galactic-nat` uplink (#673), and a `galactic-nat` metrics scrape down. For the opt-in fabric-api it adds the `fabric-router` PodMonitor's `api-metrics` endpoint (the sidecar, port `api-metrics`/`9345`, as `job=fabric-api-node`), a `fabric-api-gateway` PodMonitor (port `metrics`/`9347`) and a `fabric-api` rule group (stalled execution, node diagnostics unavailable, nodes unreachable, oversized results, cleanup backlog, certificate renewal failing, credentials missing, search index unsynced, scrape down — none fires merely because nobody ran a query). Requires the `monitoring.coreos.com` CRDs, which this repo does not install, so it is not part of the root `config/kustomization.yaml`. `task test:alerts` unit-tests the rules with `promtool` (`config/monitoring/tests/`). Every alert carries what infra's alerting standard checks for (`service: galactic`, `team: connect`, `runbook_url` into infra's `docs/runbooks/galactic/README.md`). The PrometheusRule here is the tested source, not where the rules run on Datum's infra: there the PodMonitors are converted and scraped on each edge cluster, but edge vmalert never notifies, so infra evaluates a copy of the rules centrally. Change the rules here first, with their tests, then update infra's copy. -- **`deploy/containerlab/`** — ContainerLab topology (`gvpc.clab.yaml`) for three Kind clusters (dfw, iad, sjc) wired over an SRv6 transit mesh whose underlay is dual-stack — every transit link and loopback carries an IPv4 and an IPv6 address, with one BGP session per address family (the SRv6 data plane itself stays IPv6-only; tenant IPv4 rides inside the encapsulation). Each site has one `compute` worker and at least one `edge` worker (dfw has two); every worker has its own eBGP uplink to its site's transit router, with FRR running as a hostNetwork DaemonSet. `galactic-router` (plain mode) distributes EVPN paths over iBGP to one route reflector for the whole lab (`iad-worker3`, `config/galactic-router/overlays/control/`) — compute and edge nodes never peer directly. Every `edge` node runs an egress shard (`galactic-nat`) and a `galactic-gateway`, which share the node's uplinks through its XDP dispatcher (`GALACTIC_NAT_XDP_ATTACH=dispatch`). Each site's compute node egresses only through its own site's edge shards, and each site has its own VIP `/64`, served only through its own edge nodes to one `ns60` backend in that site. A standalone off-cluster host on the fourth transit router gives tenant egress a real destination and is the client for the VIP checks (`verify:gateway-ingress`). See `deploy/containerlab/README.md` and `deploy/containerlab/Taskfile.yaml` for bring-up commands. +- **`deploy/containerlab/`** — ContainerLab topology (`gvpc.clab.yaml`) for three Kind clusters (dfw, iad, sjc) wired over an SRv6 transit mesh whose underlay is dual-stack — every transit link and loopback carries an IPv4 and an IPv6 address, with one BGP session per address family (the SRv6 data plane itself stays IPv6-only; tenant IPv4 rides inside the encapsulation). Each site has one `compute` worker and at least one `edge` worker (dfw has two); every worker has its own eBGP uplink to its site's transit router, with FRR running as a hostNetwork DaemonSet. `galactic-router` (plain mode) distributes EVPN paths over iBGP to one route reflector for the whole lab (`iad-worker3`, `config/galactic-router/overlays/control/`) — compute and edge nodes never peer directly. Every `edge` node runs an egress shard (`galactic-nat`) and a `galactic-gateway`, which share the node's uplinks through its XDP dispatcher (`GALACTIC_NAT_XDP_ATTACH=dispatch`). Each site's compute node egresses only through its own site's edge shards, and each site has its own VIP `/64`, served only through its own edge nodes to one `ns60` backend in that site. A standalone off-cluster host on the fourth transit router gives tenant egress a real destination and is the client for the VIP checks (`verify:gateway-ingress`). The lab enforces interface traffic policy (production defaults to off): every tenant binds each attachment to a grant set in `resources/tenants/policies/`, and `verify:traffic-policy` proves each path's grant and denial. See `deploy/containerlab/README.md` and `deploy/containerlab/Taskfile.yaml` for bring-up commands. ## New Developer Entry Points diff --git a/api/v1alpha1/labfixtures_test.go b/api/v1alpha1/labfixtures_test.go new file mode 100644 index 00000000..b86bdba9 --- /dev/null +++ b/api/v1alpha1/labfixtures_test.go @@ -0,0 +1,225 @@ +// Copyright 2026 Datum Cloud, Inc. +// +// SPDX-License-Identifier: AGPL-3.0-or-later + +package v1alpha1_test + +import ( + "bufio" + "bytes" + "context" + "encoding/json" + "errors" + "io" + "os" + "path/filepath" + "strings" + "testing" + + "k8s.io/apimachinery/pkg/util/validation/field" + utilyaml "k8s.io/apimachinery/pkg/util/yaml" + "sigs.k8s.io/yaml" + + galacticv1alpha1 "go.datum.net/galactic/api/v1alpha1" + "go.datum.net/galactic/internal/trafficpolicy" +) + +// labTenants is the containerlab lab's tenant fixtures, which enforce traffic +// policy (deploy/containerlab/resources/tenants/). +var labTenants = filepath.Join("..", "..", "deploy", "containerlab", "resources", "tenants") + +// yamlDocs splits every YAML document in path. +func yamlDocs(t *testing.T, path string) [][]byte { + t.Helper() + raw, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + r := utilyaml.NewYAMLReader(bufio.NewReader(bytes.NewReader(raw))) + var out [][]byte + for { + doc, err := r.Read() + if errors.Is(err, io.EOF) { + return out + } + if err != nil { + t.Fatalf("%s: %v", path, err) + } + if len(bytes.TrimSpace(doc)) > 0 { + out = append(out, doc) + } + } +} + +// labPolicies reads every policy set, by set name and policy name. +func labPolicies(t *testing.T) map[string]map[string]galacticv1alpha1.TrafficPolicy { + t.Helper() + files, err := filepath.Glob(filepath.Join(labTenants, "policies", "*", "policies.yaml")) + if err != nil || len(files) == 0 { + t.Fatalf("no lab policy sets under %s: %v", labTenants, err) + } + sets := map[string]map[string]galacticv1alpha1.TrafficPolicy{} + for _, f := range files { + set := filepath.Base(filepath.Dir(f)) + sets[set] = map[string]galacticv1alpha1.TrafficPolicy{} + for _, doc := range yamlDocs(t, f) { + var p galacticv1alpha1.TrafficPolicy + if err := yaml.UnmarshalStrict(doc, &p); err != nil { + t.Fatalf("%s: %v", f, err) + } + sets[set][p.Name] = p + } + } + return sets +} + +// TestLabPoliciesAreValid: every policy the lab enforces is admitted by the +// CRD schema and passes the webhook's Go validation and the compiler, so a +// lab deploy never stalls on a fixture the webhook refuses. +func TestLabPoliciesAreValid(t *testing.T) { + sets := labPolicies(t) + for set, policies := range sets { + for name, p := range policies { + model, errs := trafficpolicy.FromAPI(p.Spec, field.NewPath("spec")) + if len(errs) > 0 { + t.Errorf("%s/%s: %v", set, name, errs) + continue + } + if _, err := trafficpolicy.Compile(model); err != nil { + t.Errorf("%s/%s: %v", set, name, err) + } + } + } + + c, _ := startAPIServer(t) + for set, policies := range sets { + for name, p := range policies { + obj := p.DeepCopy() + obj.Name = set + "-" + name + obj.Namespace = testNamespace + if err := c.Create(context.Background(), obj); err != nil { + t.Errorf("%s/%s: the API server refused it: %v", set, name, err) + } + } + } +} + +// TestLabTenantsAreBound: every galactic attachment a lab tenant defines has +// a binding in that tenant's policy/, naming policies its set defines, so no +// lab tenant comes up denying everything. +func TestLabTenantsAreBound(t *testing.T) { + sets := labPolicies(t) + tenants, err := filepath.Glob(filepath.Join(labTenants, "ns*")) + if err != nil || len(tenants) == 0 { + t.Fatalf("no lab tenants under %s: %v", labTenants, err) + } + for _, dir := range tenants { + ns := filepath.Base(dir) + set := policySet(t, filepath.Join(dir, "policy", "kustomization.yaml")) + policies, ok := sets[set] + if !ok { + t.Errorf("%s: policy/ uses set %q, which does not exist", ns, set) + continue + } + bound := map[string]bool{} + for _, doc := range yamlDocs(t, filepath.Join(dir, "policy", "bindings.yaml")) { + var b galacticv1alpha1.TrafficPolicyBinding + if err := yaml.UnmarshalStrict(doc, &b); err != nil { + t.Fatalf("%s: %v", ns, err) + } + bound[b.Name] = true + for _, ref := range []*galacticv1alpha1.LocalPolicyReference{b.Spec.IngressPolicyRef, b.Spec.EgressPolicyRef} { + if ref == nil { + t.Errorf("%s/%s: a direction is unbound, so it denies everything", ns, b.Name) + } else if _, ok := policies[ref.Name]; !ok { + t.Errorf("%s/%s: names %q, which set %s does not define", ns, b.Name, ref.Name, set) + } + } + } + nads := map[string]bool{} + for _, nad := range galacticNADs(t, dir) { + nads[nad] = true + if !bound[nad] { + t.Errorf("%s: attachment %s has no binding in policy/bindings.yaml", ns, nad) + } + } + for name := range bound { + if !nads[name] { + t.Errorf("%s: binding %s names no attachment the tenant defines", ns, name) + } + } + } +} + +// policySet is the policy set a tenant's policy/kustomization.yaml pulls in. +// It fails unless the kustomization also applies bindings.yaml. +func policySet(t *testing.T, path string) string { + t.Helper() + var k struct { + Resources []string `json:"resources"` + } + raw, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + if err := yaml.Unmarshal(raw, &k); err != nil { + t.Fatalf("%s: %v", path, err) + } + set := "" + bindings := false + for _, r := range k.Resources { + switch { + case strings.Contains(r, "policies/"): + set = filepath.Base(r) + case r == "bindings.yaml": + bindings = true + } + } + if set == "" || !bindings { + t.Fatalf("%s must pull in one policy set and bindings.yaml, has %v", path, k.Resources) + } + return set +} + +// galacticNADs names every NetworkAttachmentDefinition under a tenant's site +// directories whose chain runs galactic-veth or galactic-tap. +func galacticNADs(t *testing.T, dir string) []string { + t.Helper() + files, err := filepath.Glob(filepath.Join(dir, "*", "*.yaml")) + if err != nil { + t.Fatal(err) + } + seen := map[string]bool{} + var out []string + for _, f := range files { + for _, doc := range yamlDocs(t, f) { + var nad struct { + Kind string `json:"kind"` + Metadata struct { + Name string `json:"name"` + } `json:"metadata"` + Spec struct { + Config string `json:"config"` + } `json:"spec"` + } + if err := yaml.Unmarshal(doc, &nad); err != nil || nad.Kind != "NetworkAttachmentDefinition" { + continue + } + var conf struct { + Plugins []struct { + Type string `json:"type"` + } `json:"plugins"` + } + if err := json.Unmarshal([]byte(nad.Spec.Config), &conf); err != nil { + t.Fatalf("%s: %s: %v", f, nad.Metadata.Name, err) + } + for _, p := range conf.Plugins { + if (p.Type == "galactic-veth" || p.Type == "galactic-tap") && !seen[nad.Metadata.Name] { + seen[nad.Metadata.Name] = true + out = append(out, nad.Metadata.Name) + } + } + } + } + return out +} diff --git a/deploy/containerlab/AGENTS.md b/deploy/containerlab/AGENTS.md index 68b8d7c6..2ab1064f 100644 --- a/deploy/containerlab/AGENTS.md +++ b/deploy/containerlab/AGENTS.md @@ -27,6 +27,8 @@ - **FRR config**: Transit router configs omit the `frr version` directive (managed by the FRR image, not the config). - **Shared manifests**: `resources/galactic-cni/kustomization.yaml` builds on `config/galactic-cni/`, `resources/galactic-router/base/kustomization.yaml` builds on `config/galactic-router/overlays/router/`, `resources/galactic-control/iad/kustomization.yaml` builds on `config/galactic-router/overlays/control/`, and `resources/fabric-router/base/kustomization.yaml` builds on `config/fabric-router/` (each in turn pulls in its own `base/` where one exists: `config/galactic-router/base/`; `config/fabric-router/` has no role split, it's a single flat DaemonSet) rather than forking them, patching in only what the lab needs to differ (image, env — for fabric, just `image: fabric-router:latest` + `imagePullPolicy: Never` on all three containers via `fabric-lab-patch.yaml`, which overrides nothing else). `config/fabric-router/`'s own affinity already covers every fabric-needing node via `galactic.datumapis.com/fabric: router` (every lab worker carries it, regardless of its other labels), so the lab inherits it as-is: one fabric DaemonSet per cluster, one `fabric-router.` ConfigMap per matching node. (A second overlay, `resources/fabric-control/iad/`, used to exist purely so the reflector node could get its own `frr.conf` back when the DaemonSet read a single shared `frr.conf` — it narrowed the base affinity to `node: [compute, edge]`, narrowed its own to `galactic: control`, and needed a JSON6902 `/metadata/name` rename to `fabric-control` to avoid clobbering the base DaemonSet in the same namespace. Per-node config selection removed the need for all three, and it was deleted; the reflector node is now served by the single DaemonSet like every other node. It was never the underlay's route reflector — that's `fabric-router-rr` on `fabric=control`, still unimplemented, see `docs/node-labels.md`.) `kubectl apply -k` refuses to load resource files from outside a kustomization's own root, so `deploy-cni.sh`/`deploy-galactic-router.sh`/`deploy-fabric.sh` `docker cp` those directories into a local `base/` subdirectory on the node at deploy time (`resources/galactic-cni/base/`, `resources/galactic-router/base/{base,overlays/router}/`, `resources/galactic-control/iad/{base,overlays/control}/` — each nested at the same relative depth as `config/galactic-router/` itself, since the copied `overlays/router`/`overlays/control` kustomization.yamls' own `../../base` references are copied verbatim, `resources/fabric-router/base/fabric/`) instead of referencing `config/` across that boundary. The `galactic-system` namespace and RBAC/ServiceAccount are applied straight from `config/galactic-system/namespace.yaml` and `config/*/{rbac,serviceaccount}.yaml` by `deploy-system.sh` via `lib.sh`'s `copy_config` (which copies all of `config/` to `/galactic/config/`) — not part of any kustomize build, so per-site `namePrefix` never touches the shared cluster-scoped RBAC. `config/galactic-system/namespace.yaml` is also the only place `galactic-system` gets created for production — there is no other bootstrap step in this repo, so if it's ever missing on a real cluster, deploying `config/galactic-router/` or `config/galactic-cni/` fails outright. +- **Traffic policy is enforced**: `resources/galactic-cni/shared/` sets `GALACTIC_CNI_TRAFFIC_POLICY=enforce` and `resources/galactic-router/base/` sets `GALACTIC_ROUTER_TRAFFIC_POLICY=true` (production defaults to off for both), so every tenant interface denies everything until galactic-router publishes its bindings. `deploy:traffic-policy` (after `deploy:cert-manager`, before `deploy:galactic-router`) installs the `galactic.datumapis.com` CRDs and `galactic-controller` (`resources/galactic-controller/`, on the host network for the same reason as cert-manager's webhook, ports 10261-10263, one replica). Each `deploy:nsNN` applies the tenant's `policy/` (one of the sets in `resources/tenants/policies/`, bound to every attachment in both directions), then its pods, then `scripts/publish-attachments.sh`, which creates any VPCAttachment the tenant does not define and writes every attachment's status, including `hostInterface` (`GH`), which the router binds by. A new tenant, NAD or check that sends a new kind of traffic needs a grant: `api/v1alpha1`'s `TestLabTenantsAreBound` fails for an unbound NAD, and `verify:traffic-policy` covers each path's grant and denial. + ## Naming Layers - **Fabric** — FRR DaemonSets running on each cluster's workers; handle eBGP to the transit mesh. Resources live in `resources/fabric-router/`. @@ -35,7 +37,7 @@ - **iad-control** — `iad-worker3`, the lab's single EVPN route reflector; every galactic-router in every cluster peers with it. galactic-router + BGP CRDs in `resources/galactic-control/iad/` (one `bgppeer-.yaml` per client), fabric in `resources/fabric-router/iad/` (`frr.conf.iad-worker3`). - **Edge** — the four `nat=enabled` workers (`dfw-worker2`, `dfw-worker3`, `sjc-worker2`, `iad-worker2`): each site's transit-facing nodes, the uplinks its compute node routes through, the egress shards and the VIP gateways. Their `BGPRouter` and route-reflector `BGPPeer` live alongside the compute node's in `resources/galactic-router//` (`bgprouter-.yaml`, `bgppeer-.yaml`). - **Egress** — `galactic-nat`, one egress shard per edge node, running from that node's XDP dispatcher on `bond0`/`bond1`. One per-site overlay under `resources/galactic-nat//` (uplink override + one `EgressShard` per edge node; dfw has two). Each site's compute node lists only its own site's shard SIDs (`resources/galactic-cni//egress-shards-patch.yaml`), and each edge node's `frr.conf` originates its shard's SID `/64` and masquerade addresses. -- **Ingress** — `galactic-gateway`, one DaemonSet per edge node (`resources/galactic-gateway//`: hostname pin, `PUBLIC_INTERFACE=bond0`, `INTERNAL_INTERFACES=bond1`, `SRV6_ADDRESS` left unset so the gateway derives the node's own uSID from its `BGPRouter`, and its `NetworkGateway`). Each site directory (`resources/galactic-gateway//`) adds the `ns60-tcp`/`ns60-udp` `NetworkRule`s for that site's own VIP, which select the site's two `ns60` backends by label through their `VPCAttachment`s (`resources/tenants/ns60//vpcattachments.yaml`, fixed addresses from their NADs; `scripts/publish-ns60-attachments.sh` writes their status, the attachment controller's job in production). The compute node's `galactic-router` writes one `ServiceVIPBinding` per rule per backend; none is hand-written. Each edge node's `frr.conf` originates its site's VIP `/64` (`2001:db8:6060:{1,2,3}::/64` for dfw, sjc, iad) to `Null0`. Both backends share the site's one compute node on purpose: the gateway carries Maglev's choice to the node in the SRv6 destination's slot, so each gets its own flows (#799). +- **Ingress** — `galactic-gateway`, one DaemonSet per edge node (`resources/galactic-gateway//`: hostname pin, `PUBLIC_INTERFACE=bond0`, `INTERNAL_INTERFACES=bond1`, `SRV6_ADDRESS` left unset so the gateway derives the node's own uSID from its `BGPRouter`, and its `NetworkGateway`). Each site directory (`resources/galactic-gateway//`) adds the `ns60-tcp`/`ns60-udp` `NetworkRule`s for that site's own VIP, which select the site's two `ns60` backends by label through their `VPCAttachment`s (`resources/tenants/ns60//vpcattachments.yaml`, fixed addresses from their NADs; `scripts/publish-attachments.sh` writes their status, as it does every tenant's, the attachment controller's job in production). The compute node's `galactic-router` writes one `ServiceVIPBinding` per rule per backend; none is hand-written. Each edge node's `frr.conf` originates its site's VIP `/64` (`2001:db8:6060:{1,2,3}::/64` for dfw, sjc, iad) to `Null0`. Both backends share the site's one compute node on purpose: the gateway carries Maglev's choice to the node in the SRv6 destination's slot, so each gets its own flows (#799). ## References diff --git a/deploy/containerlab/README.md b/deploy/containerlab/README.md index aa3c0453..716951eb 100644 --- a/deploy/containerlab/README.md +++ b/deploy/containerlab/README.md @@ -418,6 +418,7 @@ deploy/containerlab/ │ ├── galactic-nat/ # egress shards per site, one per edge node (dfw has two) │ ├── galactic-gateway/ # one gateway overlay per edge node, plus each site's │ │ # ns60 NetworkRules (backends picked by selector) +│ ├── galactic-controller/ # traffic policy's cell controller, on the host network │ └── tenants/ # test VPCs — one shared base/ (Namespace + netshoot │ ├── base/ # Deployment), each tenant patching its namespace and │ ├── ns10/ # default-network annotation; per-site dirs hold each @@ -428,9 +429,11 @@ deploy/containerlab/ │ │ # their own NAD+Deployment (distinct vpcattachment, │ │ # same vpc), not one NAD scaled to replicas: 2 — see │ │ # docs/tenants.md for why. -│ └── ns60/ # the gateway's VIP backends, two per site on one node, -│ # each with a fixed address and a VPCAttachment, -│ # answering TCP and UDP on port 80 +│ ├── ns60/ # the gateway's VIP backends, two per site on one node, +│ │ # each with a fixed address and a VPCAttachment, +│ │ # answering TCP and UDP on port 80 +│ └── policies/ # the traffic policy sets tenants bind; each +│ # tenant's own policy/ binds every attachment ├── node_files/ │ ├── dfw/ config.yaml │ ├── iad/ config.yaml @@ -450,7 +453,10 @@ deploy/containerlab/ ├── deploy-cni.sh ├── deploy-fabric.sh ├── deploy-galactic-router.sh - └── deploy-ns.sh + ├── deploy-traffic-policy.sh + ├── deploy-tenant-policy.sh + ├── deploy-ns.sh + └── publish-attachments.sh ``` ## Prerequisites @@ -458,6 +464,7 @@ deploy/containerlab/ - ContainerLab >= 0.54 - Docker - `kind` CLI +- `jq` - Host kernel with SRv6 support ## Quick start @@ -477,64 +484,83 @@ task deploy ## Tasks -| Task | Description | -| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | -| `build` | Build all container images (node, galactic-router, galactic-cni, frr, host) | -| `build:node` | Build the custom `kindest/node:galactic` image | -| `build:galactic-router` | Build the galactic-router container from Go source | -| `build:galactic-cni` | Build the galactic-cni installer image | -| `build:galactic-gateway` | Build the galactic-gateway edge load balancer image | -| `build:frr` | Build the FRR container from Alpine edge | -| `build:remote-host` | Build the off-fabric nginx host image | -| `deploy` | Build images, apply host sysctls, and deploy the lab | -| `deploy:topology` | Deploy the ContainerLab topology (transit routers) | -| `deploy:clusters` | Create the three Kind clusters and export their kubeconfigs | -| `deploy:images` | Load container images into Kind clusters | -| `deploy:system` | Install BGP and VPC CRDs; apply the galactic-system namespace and shared RBAC¹ | -| `deploy:cni` | Install Cilium and Multus, then the galactic-cni DaemonSet | -| `deploy:fabric` | Apply FRR DaemonSets to all clusters | -| `deploy:bmp-collector` | Install the BMP collector on iad-worker3, where the route reflector streams to it | -| `deploy:galactic-router` | Apply galactic-router DaemonSets and BGP CRs | -| `deploy:galactic-gateway` | Install galactic-gateway on every edge node, with each site's VIP rules (after `deploy:scenarios`) | -| `deploy:scenarios` | Deploy all VPC test scenarios | -| `deploy:ns10` | Deploy ns10 test VPC (IPv6-only, fd20 ULA) | -| `deploy:ns20` | Deploy ns20 test VPC (dual-stack, fd20 ULA + IPv4) | -| `deploy:ns30` | Deploy ns30 test VPC (dfw only, 2 pods) | -| `deploy:ns40` | Deploy ns40 test VPC (iad only, 2 pods) | -| `deploy:ns60` | Deploy ns60 test VPC (two VIP backends per site, on one node) | -| `verify:fabric-metrics` | Scrape every fabric-router's frr-exporter; fail on a failed collector or a session not Established | -| `verify:bmp` | Check the route reflector and every fabric-router stream to the BMP collector, and that it agrees with each on every session's state | -| `verify:underlay` | Ping every underlay loopback from tr1 over both IPv4 and IPv6 | -| `verify:gateway` | Check every NetworkGateway is Ready, every NetworkRule Accepted, every generated ServiceVIPBinding Bound | -| `verify:gateway-ingress` | TCP and UDP from the off-fabric host to each site's VIP, through every gateway node in turn | -| `verify:gateway-restart` | Restart dfw-worker2's gateway and shard in turn while the other carries traffic (#710) | -| `verify:gateway-detach` | Turn dfw-worker2's gateway datapath off: the ingress check must fail, then pass (not in `verify`) | -| `verify:nat-datapath` | Full egress round trip to the off-fabric host, IPv6 (NAT66) and IPv4 (NAT64) | -| `verify:nat-return-route` | Prove a reply from outside the fabric reaches each shard's masquerade addresses | -| `verify:nat-local` | Prove each site's tenants egress through that site's own edge shard | -| `verify:nat-collision` | Prove two tenants sharing a ULA on one node do not share a translation | -| `verify:mss-clamp` | Prove full-size TCP segments cross the fabric, out through NAT and across sites | -| `verify:pmtu` | Prove a packet too big for the fabric draws an ICMP error from its own node, between sites and through NAT | -| `verify:nat-icmp` | Prove ping, traceroute and path MTU discovery work through each site's egress shard | -| `verify:nat-active-active` | Hashed egress on dfw: both shards carry tenants, a failure and a drain move only theirs | -| `verify:scenarios` | Verify ping across all VPC test scenarios | -| `verify:ns10` | Verify ns10 ping (IPv6-only, 3-site mesh) | -| `verify:ns20` | Verify ns20 ping (dual-stack, 3-site mesh) | -| `verify:ns30` | Verify ns30 ping (dfw only, 2 pods) | -| `verify:ns40` | Verify ns40 ping (iad only, 2 pods) | -| `destroy` | Destroy the lab and remove all Kind clusters | -| `restart` | Full rebuild — destroy then redeploy | -| `rebuild` | Full rebuild — clean (destroy + delete images/artifacts) then redeploy | -| `inspect` | Show running nodes and management addresses | -| `graph` | Generate a draw.io diagram for the topology | -| `host-setup` | Apply required host sysctls (IPv6 forwarding, inotify limits) | -| `clean` | Destroy lab, delete built images, and remove lab artifacts | -| `test` | Run all verification checks | +| Task | Description | +| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | +| `build` | Build all container images (node, galactic-router, galactic-cni, frr, host) | +| `build:node` | Build the custom `kindest/node:galactic` image | +| `build:galactic-router` | Build the galactic-router container from Go source | +| `build:galactic-cni` | Build the galactic-cni installer image | +| `build:galactic-gateway` | Build the galactic-gateway edge load balancer image | +| `build:frr` | Build the FRR container from Alpine edge | +| `build:remote-host` | Build the off-fabric nginx host image | +| `build:galactic-controller` | Build the galactic-controller image (traffic policy's cell role) | +| `deploy` | Build images, apply host sysctls, and deploy the lab | +| `deploy:topology` | Deploy the ContainerLab topology (transit routers) | +| `deploy:clusters` | Create the three Kind clusters and export their kubeconfigs | +| `deploy:images` | Load container images into Kind clusters | +| `deploy:system` | Install BGP and VPC CRDs; apply the galactic-system namespace and shared RBAC¹ | +| `deploy:cni` | Install Cilium and Multus, then the galactic-cni DaemonSet | +| `deploy:traffic-policy` | Install the traffic policy CRDs and galactic-controller, and wait for its webhook (after `deploy:cert-manager`) | +| `deploy:fabric` | Apply FRR DaemonSets to all clusters | +| `deploy:bmp-collector` | Install the BMP collector on iad-worker3, where the route reflector streams to it | +| `deploy:galactic-router` | Apply galactic-router DaemonSets and BGP CRs | +| `deploy:galactic-gateway` | Install galactic-gateway on every edge node, with each site's VIP rules (after `deploy:scenarios`) | +| `deploy:scenarios` | Deploy all VPC test scenarios | +| `deploy:ns10` | Deploy ns10 test VPC (IPv6-only, fd20 ULA) | +| `deploy:ns20` | Deploy ns20 test VPC (dual-stack, fd20 ULA + IPv4) | +| `deploy:ns30` | Deploy ns30 test VPC (dfw only, 2 pods) | +| `deploy:ns40` | Deploy ns40 test VPC (iad only, 2 pods) | +| `deploy:ns60` | Deploy ns60 test VPC (two VIP backends per site, on one node) | +| `verify:fabric-metrics` | Scrape every fabric-router's frr-exporter; fail on a failed collector or a session not Established | +| `verify:bmp` | Check the route reflector and every fabric-router stream to the BMP collector, and that it agrees with each on every session's state | +| `verify:underlay` | Ping every underlay loopback from tr1 over both IPv4 and IPv6 | +| `verify:gateway` | Check every NetworkGateway is Ready, every NetworkRule Accepted, every generated ServiceVIPBinding Bound | +| `verify:gateway-ingress` | TCP and UDP from the off-fabric host to each site's VIP, through every gateway node in turn | +| `verify:gateway-restart` | Restart dfw-worker2's gateway and shard in turn while the other carries traffic (#710) | +| `verify:gateway-detach` | Turn dfw-worker2's gateway datapath off: the ingress check must fail, then pass (not in `verify`) | +| `verify:nat-datapath` | Full egress round trip to the off-fabric host, IPv6 (NAT66) and IPv4 (NAT64) | +| `verify:nat-return-route` | Prove a reply from outside the fabric reaches each shard's masquerade addresses | +| `verify:nat-local` | Prove each site's tenants egress through that site's own edge shard | +| `verify:nat-collision` | Prove two tenants sharing a ULA on one node do not share a translation | +| `verify:mss-clamp` | Prove full-size TCP segments cross the fabric, out through NAT and across sites | +| `verify:pmtu` | Prove a packet too big for the fabric draws an ICMP error from its own node, between sites and through NAT | +| `verify:nat-icmp` | Prove ping, traceroute and path MTU discovery work through each site's egress shard | +| `verify:nat-active-active` | Hashed egress on dfw: both shards carry tenants, a failure and a drain move only theirs | +| `verify:scenarios` | Verify ping across all VPC test scenarios | +| `verify:ns10` | Verify ns10 ping (IPv6-only, 3-site mesh) | +| `verify:ns20` | Verify ns20 ping (dual-stack, 3-site mesh) | +| `verify:ns30` | Verify ns30 ping (dfw only, 2 pods) | +| `verify:ns40` | Verify ns40 ping (iad only, 2 pods) | +| `verify:traffic-policy` | Every binding installed; each path forwards what it grants and drops, and counts, what it does not | +| `destroy` | Destroy the lab and remove all Kind clusters | +| `restart` | Full rebuild — destroy then redeploy | +| `rebuild` | Full rebuild — clean (destroy + delete images/artifacts) then redeploy | +| `inspect` | Show running nodes and management addresses | +| `graph` | Generate a draw.io diagram for the topology | +| `host-setup` | Apply required host sysctls (IPv6 forwarding, inotify limits) | +| `clean` | Destroy lab, delete built images, and remove lab artifacts | +| `test` | Run all verification checks | ¹ Network CRDs come from the latest commit on `datum-cloud/network`'s `main`, not the version `go.mod` requires. Set `NETWORK_REF=` to track a different branch, or `NETWORK_SHA=` to pin one commit. +## Traffic policy + +The lab enforces interface traffic policy (`GALACTIC_CNI_TRAFFIC_POLICY=enforce` +in `resources/galactic-cni/shared/`), while production defaults to `off`. A +tenant interface therefore forwards nothing until galactic-router publishes +its bindings, so each `deploy:nsNN` task applies the tenant's policies and +bindings (`scripts/deploy-tenant-policy.sh`, from +`resources/tenants//policy/`), deploys its pods, then publishes every +attachment's status, including the host interface the bindings attach to +(`scripts/publish-attachments.sh`, which needs `jq` on the host). The policy +sets in `resources/tenants/policies/` grant each tenant only the paths its +checks use, so a new check that sends something new needs a grant too. +`task verify:traffic-policy` proves a grant and a denial on the Private, +PublicNAT and PublicService paths and the built-in default. See +[docs/traffic-policy/rollout.md](../../docs/traffic-policy/rollout.md). + ## Verification See [docs/verification.md](docs/verification.md) for transit fabric, FRR, and galactic-router diff --git a/deploy/containerlab/Taskfile.yaml b/deploy/containerlab/Taskfile.yaml index 1081de51..3152aa67 100644 --- a/deploy/containerlab/Taskfile.yaml +++ b/deploy/containerlab/Taskfile.yaml @@ -23,6 +23,7 @@ tasks: - "build:galactic-cni" - "build:galactic-nat" - "build:galactic-gateway" + - "build:galactic-controller" - "build:remote-host" "build:node": @@ -72,6 +73,11 @@ tasks: cmds: - docker build --network=host --build-context network=../../../network -t galactic-gateway:latest -f ../../containers/galactic-gateway/Dockerfile ../.. + "build:galactic-controller": + desc: Build the galactic-controller image (traffic policy's cell role) + cmds: + - docker build --network=host --build-context network=../../../network -t galactic-controller:latest -f ../../containers/galactic-controller/Dockerfile ../.. + "build:remote-host": desc: Build the off-fabric remote host image (nginx) cmds: @@ -95,6 +101,10 @@ tasks: # mounts a csi-driver volume, and without the driver the pod (FRR # included) cannot start. - task: "deploy:cert-manager" + # After cert-manager, which issues its webhook's certificate, and + # before deploy:galactic-router, whose traffic policy reconciler + # watches the CRDs it installs. + - task: "deploy:traffic-policy" - task: "deploy:fabric" - task: "deploy:bmp-collector" - task: "deploy:galactic-router" @@ -263,6 +273,8 @@ tasks: load galactic-cni:latest {{.CNI_NODES}} load galactic-nat:latest {{.NAT_NODES}} load galactic-gateway:latest {{.GATEWAY_NODES}} + # The cell controller schedules on any untainted worker. + load galactic-controller:latest {{.FABRIC_NODES}} "deploy:system": desc: Install CRDs, namespace, and RBAC @@ -302,6 +314,11 @@ tasks: cmds: - ./scripts/deploy-fabric.sh + "deploy:traffic-policy": + desc: Install the traffic policy CRDs and galactic-controller in every cluster + cmds: + - ./scripts/deploy-traffic-policy.sh + "deploy:cert-manager": desc: Install cert-manager and its csi-driver (fabric-api certificates) cmds: @@ -345,33 +362,44 @@ tasks: "deploy:ns10": desc: Deploy ns10 test VPC (IPv6-only, fd20 ULA) cmds: + - ./scripts/deploy-tenant-policy.sh ns10 dfw sjc iad - ./scripts/deploy-ns.sh ns10 dfw sjc iad + - ./scripts/publish-attachments.sh ns10 dfw sjc iad "deploy:ns20": desc: Deploy ns20 test VPC (dual-stack) cmds: + - ./scripts/deploy-tenant-policy.sh ns20 dfw sjc iad - ./scripts/deploy-ns.sh ns20 dfw sjc iad + - ./scripts/publish-attachments.sh ns20 dfw sjc iad "deploy:ns30": desc: Deploy ns30 test VPC (dfw only, 2 pods) cmds: + - ./scripts/deploy-tenant-policy.sh ns30 dfw - ./scripts/deploy-ns.sh ns30 dfw + - ./scripts/publish-attachments.sh ns30 dfw "deploy:ns40": desc: Deploy ns40 test VPC (iad only, 2 pods) cmds: + - ./scripts/deploy-tenant-policy.sh ns40 iad - ./scripts/deploy-ns.sh ns40 iad + - ./scripts/publish-attachments.sh ns40 iad "deploy:ns60": desc: Deploy ns60 test VPC (two VIP backends per site, on one node) cmds: + - ./scripts/deploy-tenant-policy.sh ns60 dfw sjc iad - ./scripts/deploy-ns.sh ns60 dfw sjc iad - - ./scripts/publish-ns60-attachments.sh dfw sjc iad + - ./scripts/publish-attachments.sh ns60 dfw sjc iad "deploy:ns71": desc: Deploy ns71 test VPCs (overlapping ULA on one node, same-node collision fixture) cmds: + - ./scripts/deploy-tenant-policy.sh ns71 sjc - ./scripts/deploy-ns.sh ns71 sjc + - ./scripts/publish-attachments.sh ns71 sjc verify: desc: Run all verification checks @@ -400,6 +428,9 @@ tasks: - task: "verify:nat64-non-global" - task: "verify:nat-active-active" - task: "verify:scenarios" + # Last: it briefly rebinds and edits tenant policies, and restores + # them before it returns. + - task: "verify:traffic-policy" # First, because every check after it rides these bonds: a member that # never joined its aggregate leaves everything else passing over the @@ -1105,6 +1136,13 @@ tasks: - task: "verify:ns30" - task: "verify:ns40" + # Not idempotent while it runs: it edits ns60's dfw policy and unbinds + # ns30/private-b's ingress for a moment, and restores both on exit. + "verify:traffic-policy": + desc: Verify every tenant binding is installed and each path forwards what it grants and drops what it does not + cmds: + - ./scripts/verify-traffic-policy.sh + "verify:ns10": desc: Verify ns10 ping (IPv6-only, 3-site mesh) cmds: diff --git a/deploy/containerlab/docs/tenants.md b/deploy/containerlab/docs/tenants.md index b2262269..c57a3155 100644 --- a/deploy/containerlab/docs/tenants.md +++ b/deploy/containerlab/docs/tenants.md @@ -17,6 +17,12 @@ for the full per-binary ADD sequence). The `galactic-router` controller then advertises each pod's EVPN route to the route reflector, distributing reachability across sites. +The lab enforces interface traffic policy, so each tenant also binds every +attachment to a policy set in both directions (`resources/tenants//policy/`, +see `resources/tenants/policies/README.md`), and `deploy:nsNN` publishes each +attachment's status after its pods start (`scripts/publish-attachments.sh`). A +pod's interface forwards nothing until then. + They differ only in scope and addressing: | VPC | Sites | Address families | VRF interface | Notes | diff --git a/deploy/containerlab/resources/galactic-cni/shared/daemonset-patch.yaml b/deploy/containerlab/resources/galactic-cni/shared/daemonset-patch.yaml index 7d19cbfc..0f3701cc 100644 --- a/deploy/containerlab/resources/galactic-cni/shared/daemonset-patch.yaml +++ b/deploy/containerlab/resources/galactic-cni/shared/daemonset-patch.yaml @@ -26,6 +26,13 @@ spec: image: galactic-cni:latest imagePullPolicy: Never env: + # The lab enforces interface traffic policy, though production + # defaults to off: every tenant interface denies until + # galactic-router publishes its bindings, and each tenant's + # fixtures grant what the lab's checks send + # (resources/tenants/policies/, verify:traffic-policy). + - name: GALACTIC_CNI_TRAFFIC_POLICY + value: enforce # Fabric uplink the eBPF hook attaches to - name: GALACTIC_CNI_EBPF_INTERFACES value: bond0 diff --git a/deploy/containerlab/resources/galactic-controller/controller-lab-patch.yaml b/deploy/containerlab/resources/galactic-controller/controller-lab-patch.yaml new file mode 100644 index 00000000..a1616cc1 --- /dev/null +++ b/deploy/containerlab/resources/galactic-controller/controller-lab-patch.yaml @@ -0,0 +1,29 @@ +# The lab's control-plane node cannot reach pod addresses on the workers over +# Cilium's tunnel, so the kube-apiserver could never call a pod-network +# webhook; it reaches a worker's own address over the Kind network, as +# cert-manager's webhook does (scripts/deploy-cert-manager.sh). The webhook +# fails closed, so an unreachable one would refuse every TrafficPolicy. +# +# On the host network the ports must not collide with anything else on a +# worker: cert-manager's webhook holds 10260, and galactic-gateway serves its +# metrics on 8081, the base's health port. One replica: iad has a single +# untainted worker, where a second host-network replica could never bind. For +# the same reason a rolling update could never place its surge pod, which +# needs the ports the old one holds, so the pod is recreated instead. +apiVersion: apps/v1 +kind: Deployment +metadata: + name: galactic-controller + namespace: galactic-system +spec: + replicas: 1 + strategy: + type: Recreate + template: + spec: + hostNetwork: true + dnsPolicy: ClusterFirstWithHostNet + containers: + - name: controller + image: galactic-controller:latest + imagePullPolicy: Never diff --git a/deploy/containerlab/resources/galactic-controller/kustomization.yaml b/deploy/containerlab/resources/galactic-controller/kustomization.yaml new file mode 100644 index 00000000..20e8d774 --- /dev/null +++ b/deploy/containerlab/resources/galactic-controller/kustomization.yaml @@ -0,0 +1,42 @@ +# The lab's galactic-controller --role=cell: config/galactic-controller/base, +# copied into base/ at deploy time (scripts/deploy-traffic-policy.sh), with +# only what the lab needs to differ patched in (see +# controller-lab-patch.yaml for why). +resources: + - base +patches: + - path: controller-lab-patch.yaml + # With one replica, the base's minAvailable: 1 budget would block draining + # the worker it runs on. + - patch: |- + $patch: delete + apiVersion: policy/v1 + kind: PodDisruptionBudget + metadata: + name: galactic-controller + namespace: galactic-system + # Replaced rather than merged: a strategic merge keys ports on + # containerPort and would keep the base's alongside these. + - target: + kind: Deployment + name: galactic-controller + patch: |- + - op: replace + path: /spec/template/spec/containers/0/args + value: + - --role=cell + - --webhook-port=10261 + - --health-probe-bind-address=:10262 + - --metrics-bind-address=:10263 + - op: replace + path: /spec/template/spec/containers/0/ports + value: + - name: webhook + containerPort: 10261 + protocol: TCP + - name: health + containerPort: 10262 + protocol: TCP + - name: metrics + containerPort: 10263 + protocol: TCP diff --git a/deploy/containerlab/resources/galactic-router/base/router-lab-patch.yaml b/deploy/containerlab/resources/galactic-router/base/router-lab-patch.yaml index 8675980a..ca97424b 100644 --- a/deploy/containerlab/resources/galactic-router/base/router-lab-patch.yaml +++ b/deploy/containerlab/resources/galactic-router/base/router-lab-patch.yaml @@ -9,3 +9,11 @@ spec: - name: galactic-router image: galactic-router:latest imagePullPolicy: Never + env: + # The lab enforces interface traffic policy + # (resources/galactic-cni/shared/), so every router publishes + # its node's bindings. deploy:traffic-policy installs the CRDs + # this watches first. Only this plain role: the reflector has + # no tenant interfaces and refuses it. + - name: GALACTIC_ROUTER_TRAFFIC_POLICY + value: "true" diff --git a/deploy/containerlab/resources/tenants/ns10/policy/bindings.yaml b/deploy/containerlab/resources/tenants/ns10/policy/bindings.yaml new file mode 100644 index 00000000..75eb70c6 --- /dev/null +++ b/deploy/containerlab/resources/tenants/ns10/policy/bindings.yaml @@ -0,0 +1,9 @@ +apiVersion: galactic.datumapis.com/v1alpha1 +kind: TrafficPolicyBinding +metadata: + name: private +spec: + ingressPolicyRef: + name: lab-ingress + egressPolicyRef: + name: lab-egress diff --git a/deploy/containerlab/resources/tenants/ns10/policy/kustomization.yaml b/deploy/containerlab/resources/tenants/ns10/policy/kustomization.yaml new file mode 100644 index 00000000..cf4d7eaf --- /dev/null +++ b/deploy/containerlab/resources/tenants/ns10/policy/kustomization.yaml @@ -0,0 +1,7 @@ +# ns10's traffic policy: the private-nat set (../../policies/README.md), bound to +# every attachment ns10 has, in both directions. A binding is named after +# its VPCAttachment, which scripts/publish-attachments.sh creates. +namespace: ns10 +resources: + - ../../policies/private-nat + - bindings.yaml diff --git a/deploy/containerlab/resources/tenants/ns20/policy/bindings.yaml b/deploy/containerlab/resources/tenants/ns20/policy/bindings.yaml new file mode 100644 index 00000000..75eb70c6 --- /dev/null +++ b/deploy/containerlab/resources/tenants/ns20/policy/bindings.yaml @@ -0,0 +1,9 @@ +apiVersion: galactic.datumapis.com/v1alpha1 +kind: TrafficPolicyBinding +metadata: + name: private +spec: + ingressPolicyRef: + name: lab-ingress + egressPolicyRef: + name: lab-egress diff --git a/deploy/containerlab/resources/tenants/ns20/policy/kustomization.yaml b/deploy/containerlab/resources/tenants/ns20/policy/kustomization.yaml new file mode 100644 index 00000000..c37ff9c4 --- /dev/null +++ b/deploy/containerlab/resources/tenants/ns20/policy/kustomization.yaml @@ -0,0 +1,7 @@ +# ns20's traffic policy: the private-nat set (../../policies/README.md), bound to +# every attachment ns20 has, in both directions. A binding is named after +# its VPCAttachment, which scripts/publish-attachments.sh creates. +namespace: ns20 +resources: + - ../../policies/private-nat + - bindings.yaml diff --git a/deploy/containerlab/resources/tenants/ns30/policy/bindings.yaml b/deploy/containerlab/resources/tenants/ns30/policy/bindings.yaml new file mode 100644 index 00000000..9d8f8e9b --- /dev/null +++ b/deploy/containerlab/resources/tenants/ns30/policy/bindings.yaml @@ -0,0 +1,19 @@ +apiVersion: galactic.datumapis.com/v1alpha1 +kind: TrafficPolicyBinding +metadata: + name: private +spec: + ingressPolicyRef: + name: lab-ingress + egressPolicyRef: + name: lab-egress +--- +apiVersion: galactic.datumapis.com/v1alpha1 +kind: TrafficPolicyBinding +metadata: + name: private-b +spec: + ingressPolicyRef: + name: lab-ingress + egressPolicyRef: + name: lab-egress diff --git a/deploy/containerlab/resources/tenants/ns30/policy/kustomization.yaml b/deploy/containerlab/resources/tenants/ns30/policy/kustomization.yaml new file mode 100644 index 00000000..8611e374 --- /dev/null +++ b/deploy/containerlab/resources/tenants/ns30/policy/kustomization.yaml @@ -0,0 +1,7 @@ +# ns30's traffic policy: the private-nat set (../../policies/README.md), bound to +# every attachment ns30 has, in both directions. A binding is named after +# its VPCAttachment, which scripts/publish-attachments.sh creates. +namespace: ns30 +resources: + - ../../policies/private-nat + - bindings.yaml diff --git a/deploy/containerlab/resources/tenants/ns40/policy/bindings.yaml b/deploy/containerlab/resources/tenants/ns40/policy/bindings.yaml new file mode 100644 index 00000000..9d8f8e9b --- /dev/null +++ b/deploy/containerlab/resources/tenants/ns40/policy/bindings.yaml @@ -0,0 +1,19 @@ +apiVersion: galactic.datumapis.com/v1alpha1 +kind: TrafficPolicyBinding +metadata: + name: private +spec: + ingressPolicyRef: + name: lab-ingress + egressPolicyRef: + name: lab-egress +--- +apiVersion: galactic.datumapis.com/v1alpha1 +kind: TrafficPolicyBinding +metadata: + name: private-b +spec: + ingressPolicyRef: + name: lab-ingress + egressPolicyRef: + name: lab-egress diff --git a/deploy/containerlab/resources/tenants/ns40/policy/kustomization.yaml b/deploy/containerlab/resources/tenants/ns40/policy/kustomization.yaml new file mode 100644 index 00000000..2bf048c4 --- /dev/null +++ b/deploy/containerlab/resources/tenants/ns40/policy/kustomization.yaml @@ -0,0 +1,7 @@ +# ns40's traffic policy: the private set (../../policies/README.md), bound to +# every attachment ns40 has, in both directions. A binding is named after +# its VPCAttachment, which scripts/publish-attachments.sh creates. +namespace: ns40 +resources: + - ../../policies/private + - bindings.yaml diff --git a/deploy/containerlab/resources/tenants/ns60/dfw/vpcattachments.yaml b/deploy/containerlab/resources/tenants/ns60/dfw/vpcattachments.yaml index 16088656..6e308ef2 100644 --- a/deploy/containerlab/resources/tenants/ns60/dfw/vpcattachments.yaml +++ b/deploy/containerlab/resources/tenants/ns60/dfw/vpcattachments.yaml @@ -3,7 +3,7 @@ # In production the attachment controller writes these and their status. The # lab has no such controller, so deploy:ns60 applies them here and then # writes each one's status from where its pod actually runs -# (scripts/publish-ns60-attachments.sh). +# (scripts/publish-attachments.sh). apiVersion: cloud.datumapis.com/v1alpha1 kind: VPCAttachment metadata: diff --git a/deploy/containerlab/resources/tenants/ns60/iad/vpcattachments.yaml b/deploy/containerlab/resources/tenants/ns60/iad/vpcattachments.yaml index 177906a5..a4764999 100644 --- a/deploy/containerlab/resources/tenants/ns60/iad/vpcattachments.yaml +++ b/deploy/containerlab/resources/tenants/ns60/iad/vpcattachments.yaml @@ -3,7 +3,7 @@ # In production the attachment controller writes these and their status. The # lab has no such controller, so deploy:ns60 applies them here and then # writes each one's status from where its pod actually runs -# (scripts/publish-ns60-attachments.sh). +# (scripts/publish-attachments.sh). apiVersion: cloud.datumapis.com/v1alpha1 kind: VPCAttachment metadata: diff --git a/deploy/containerlab/resources/tenants/ns60/policy/bindings.yaml b/deploy/containerlab/resources/tenants/ns60/policy/bindings.yaml new file mode 100644 index 00000000..b4ae20c3 --- /dev/null +++ b/deploy/containerlab/resources/tenants/ns60/policy/bindings.yaml @@ -0,0 +1,19 @@ +apiVersion: galactic.datumapis.com/v1alpha1 +kind: TrafficPolicyBinding +metadata: + name: backend-a +spec: + ingressPolicyRef: + name: lab-ingress + egressPolicyRef: + name: lab-egress +--- +apiVersion: galactic.datumapis.com/v1alpha1 +kind: TrafficPolicyBinding +metadata: + name: backend-b +spec: + ingressPolicyRef: + name: lab-ingress + egressPolicyRef: + name: lab-egress diff --git a/deploy/containerlab/resources/tenants/ns60/policy/kustomization.yaml b/deploy/containerlab/resources/tenants/ns60/policy/kustomization.yaml new file mode 100644 index 00000000..8bd38cfd --- /dev/null +++ b/deploy/containerlab/resources/tenants/ns60/policy/kustomization.yaml @@ -0,0 +1,7 @@ +# ns60's traffic policy: the public-service set (../../policies/README.md), bound to +# every attachment ns60 has, in both directions. A binding is named after +# its VPCAttachment, which scripts/publish-attachments.sh creates. +namespace: ns60 +resources: + - ../../policies/public-service + - bindings.yaml diff --git a/deploy/containerlab/resources/tenants/ns60/sjc/vpcattachments.yaml b/deploy/containerlab/resources/tenants/ns60/sjc/vpcattachments.yaml index 835edea5..95a13e55 100644 --- a/deploy/containerlab/resources/tenants/ns60/sjc/vpcattachments.yaml +++ b/deploy/containerlab/resources/tenants/ns60/sjc/vpcattachments.yaml @@ -3,7 +3,7 @@ # In production the attachment controller writes these and their status. The # lab has no such controller, so deploy:ns60 applies them here and then # writes each one's status from where its pod actually runs -# (scripts/publish-ns60-attachments.sh). +# (scripts/publish-attachments.sh). apiVersion: cloud.datumapis.com/v1alpha1 kind: VPCAttachment metadata: diff --git a/deploy/containerlab/resources/tenants/ns71/policy/bindings.yaml b/deploy/containerlab/resources/tenants/ns71/policy/bindings.yaml new file mode 100644 index 00000000..9d8f8e9b --- /dev/null +++ b/deploy/containerlab/resources/tenants/ns71/policy/bindings.yaml @@ -0,0 +1,19 @@ +apiVersion: galactic.datumapis.com/v1alpha1 +kind: TrafficPolicyBinding +metadata: + name: private +spec: + ingressPolicyRef: + name: lab-ingress + egressPolicyRef: + name: lab-egress +--- +apiVersion: galactic.datumapis.com/v1alpha1 +kind: TrafficPolicyBinding +metadata: + name: private-b +spec: + ingressPolicyRef: + name: lab-ingress + egressPolicyRef: + name: lab-egress diff --git a/deploy/containerlab/resources/tenants/ns71/policy/kustomization.yaml b/deploy/containerlab/resources/tenants/ns71/policy/kustomization.yaml new file mode 100644 index 00000000..f83178ac --- /dev/null +++ b/deploy/containerlab/resources/tenants/ns71/policy/kustomization.yaml @@ -0,0 +1,7 @@ +# ns71's traffic policy: the private-nat set (../../policies/README.md), bound to +# every attachment ns71 has, in both directions. A binding is named after +# its VPCAttachment, which scripts/publish-attachments.sh creates. +namespace: ns71 +resources: + - ../../policies/private-nat + - bindings.yaml diff --git a/deploy/containerlab/resources/tenants/policies/README.md b/deploy/containerlab/resources/tenants/policies/README.md new file mode 100644 index 00000000..6ee26fee --- /dev/null +++ b/deploy/containerlab/resources/tenants/policies/README.md @@ -0,0 +1,17 @@ +# Lab traffic policies + +The lab enforces interface traffic policy (`GALACTIC_CNI_TRAFFIC_POLICY=enforce` +in `resources/galactic-cni/shared/`), so a tenant interface forwards only what +its bindings grant. These are the grants, one set per kind of tenant, each an +ingress and an egress policy granting only the paths that tenant's checks use: + +| Set | Tenants | Egress grants | Ingress grants | +| ---------------- | ---------------------- | ------------------ | --------------------------------- | +| `private-nat` | ns10, ns20, ns30, ns71 | Private, PublicNAT | Private (except TCP 9), PublicNAT | +| `private` | ns40 | Private | Private (except TCP 9) | +| `public-service` | ns60 | PublicService | PublicService | + +The TCP 9 deny on Private ingress, and every path a set leaves out, are what +`verify:traffic-policy` uses to prove a denial is enforced and counted. Each +tenant's `/policy/` applies one set into its namespace and binds every +attachment the tenant has to it, in both directions. diff --git a/deploy/containerlab/resources/tenants/policies/private-nat/kustomization.yaml b/deploy/containerlab/resources/tenants/policies/private-nat/kustomization.yaml new file mode 100644 index 00000000..1bf9e66a --- /dev/null +++ b/deploy/containerlab/resources/tenants/policies/private-nat/kustomization.yaml @@ -0,0 +1,2 @@ +resources: + - policies.yaml diff --git a/deploy/containerlab/resources/tenants/policies/private-nat/policies.yaml b/deploy/containerlab/resources/tenants/policies/private-nat/policies.yaml new file mode 100644 index 00000000..5f9079f4 --- /dev/null +++ b/deploy/containerlab/resources/tenants/policies/private-nat/policies.yaml @@ -0,0 +1,47 @@ +# Tenants that talk privately across the fabric and egress through the NAT +# shards: ns10 and ns20 (cross-site pings, NAT66/NAT64 egress, MSS clamping, +# path MTU errors from the node), ns30 (two attachments on dfw's compute +# node, which verify:nat-active-active spreads across dfw's shards with the +# rest) and ns71 (the NAT collision fixture). +apiVersion: galactic.datumapis.com/v1alpha1 +kind: TrafficPolicy +metadata: + name: lab-egress +spec: + rules: + - name: allow-private + action: Allow + match: + path: Private + - name: allow-nat + action: Allow + match: + path: PublicNAT +--- +apiVersion: galactic.datumapis.com/v1alpha1 +kind: TrafficPolicy +metadata: + name: lab-ingress +spec: + rules: + # verify:traffic-policy's negative check on the Private path: a TCP + # listener on port 9 is never reached, while the same listener on another + # port is. + - name: deny-discard + action: Deny + match: + path: Private + protocol: TCP + destinationPorts: + - start: 9 + end: 9 + # Also what the node itself sends the workload, such as a Packet Too Big + # (verify:pmtu). + - name: allow-private + action: Allow + match: + path: Private + - name: allow-nat-returns + action: Allow + match: + path: PublicNAT diff --git a/deploy/containerlab/resources/tenants/policies/private/kustomization.yaml b/deploy/containerlab/resources/tenants/policies/private/kustomization.yaml new file mode 100644 index 00000000..1bf9e66a --- /dev/null +++ b/deploy/containerlab/resources/tenants/policies/private/kustomization.yaml @@ -0,0 +1,2 @@ +resources: + - policies.yaml diff --git a/deploy/containerlab/resources/tenants/policies/private/policies.yaml b/deploy/containerlab/resources/tenants/policies/private/policies.yaml new file mode 100644 index 00000000..043abd3a --- /dev/null +++ b/deploy/containerlab/resources/tenants/policies/private/policies.yaml @@ -0,0 +1,31 @@ +# Tenants that only talk privately: ns40 (two attachments on one node). +# Nothing here grants PublicNAT, so its egress through a NAT shard is denied. +apiVersion: galactic.datumapis.com/v1alpha1 +kind: TrafficPolicy +metadata: + name: lab-egress +spec: + rules: + - name: allow-private + action: Allow + match: + path: Private +--- +apiVersion: galactic.datumapis.com/v1alpha1 +kind: TrafficPolicy +metadata: + name: lab-ingress +spec: + rules: + - name: deny-discard + action: Deny + match: + path: Private + protocol: TCP + destinationPorts: + - start: 9 + end: 9 + - name: allow-private + action: Allow + match: + path: Private diff --git a/deploy/containerlab/resources/tenants/policies/public-service/kustomization.yaml b/deploy/containerlab/resources/tenants/policies/public-service/kustomization.yaml new file mode 100644 index 00000000..1bf9e66a --- /dev/null +++ b/deploy/containerlab/resources/tenants/policies/public-service/kustomization.yaml @@ -0,0 +1,2 @@ +resources: + - policies.yaml diff --git a/deploy/containerlab/resources/tenants/policies/public-service/policies.yaml b/deploy/containerlab/resources/tenants/policies/public-service/policies.yaml new file mode 100644 index 00000000..b5ea1c18 --- /dev/null +++ b/deploy/containerlab/resources/tenants/policies/public-service/policies.yaml @@ -0,0 +1,25 @@ +# ns60's VIP backends: they receive the gateways' public service traffic and +# answer it, and nothing else (verify:gateway-ingress). verify:traffic-policy +# inserts a temporary deny ahead of allow-service to prove a denial on this +# path, then restores this policy. +apiVersion: galactic.datumapis.com/v1alpha1 +kind: TrafficPolicy +metadata: + name: lab-egress +spec: + rules: + - name: allow-service-replies + action: Allow + match: + path: PublicService +--- +apiVersion: galactic.datumapis.com/v1alpha1 +kind: TrafficPolicy +metadata: + name: lab-ingress +spec: + rules: + - name: allow-service + action: Allow + match: + path: PublicService diff --git a/deploy/containerlab/scripts/deploy-tenant-policy.sh b/deploy/containerlab/scripts/deploy-tenant-policy.sh new file mode 100755 index 00000000..7c6312ef --- /dev/null +++ b/deploy/containerlab/scripts/deploy-tenant-policy.sh @@ -0,0 +1,34 @@ +#!/bin/bash +# deploy-tenant-policy.sh — apply a tenant's traffic policies and bindings +# (resources/tenants//policy/) to the given sites. +# +# The lab enforces traffic policy, so a tenant's interfaces deny everything +# until these are bound and galactic-router publishes them. Runs before the +# tenant's pods, so a pod's interface is granted as soon as its attachment is +# published (scripts/publish-attachments.sh). +# +# Usage: deploy-tenant-policy.sh [site...] +set -euo pipefail + +if [ "$#" -lt 2 ]; then + echo "usage: $(basename "$0") [site...]" >&2 + exit 1 +fi + +NS="$1" +shift + +SCRIPT_DIR=$(cd "$(dirname "$0")" && pwd) +source "${SCRIPT_DIR}/lib.sh" + +for site in "$@"; do + node=$(control_plane "${site}") + echo "Applying ${NS}'s traffic policy to ${site}..." + ensure_namespace "${node}" "${NS}" + # The policy sets each tenant's policy/ kustomization pulls in. + copy_to "${node}" tenants/policies + copy_to "${node}" "tenants/${NS}/policy" + apply_k "${node}" "/galactic/resources/tenants/${NS}/policy/" +done + +echo "Done." diff --git a/deploy/containerlab/scripts/deploy-traffic-policy.sh b/deploy/containerlab/scripts/deploy-traffic-policy.sh new file mode 100755 index 00000000..9fe893c9 --- /dev/null +++ b/deploy/containerlab/scripts/deploy-traffic-policy.sh @@ -0,0 +1,67 @@ +#!/bin/bash +# deploy-traffic-policy.sh — install interface traffic policy's cell side in +# every cluster: the galactic.datumapis.com CRDs and galactic-controller +# --role=cell (its ServiceAccount, RBAC, and the lab overlay of +# config/galactic-controller/base), then wait until its fail-closed webhook +# admits a policy. +# +# Runs after deploy:cert-manager, which issues the webhook's serving +# certificate, and before deploy:galactic-router, whose traffic policy +# reconciler (GALACTIC_ROUTER_TRAFFIC_POLICY) watches these CRDs and never +# syncs without them. Each tenant's own policies and bindings are applied +# with the tenant (scripts/deploy-tenant-policy.sh). +set -euo pipefail + +SCRIPT_DIR=$(cd "$(dirname "$0")" && pwd) +source "${SCRIPT_DIR}/lib.sh" + +CONFIG=$(cd "${SCRIPT_DIR}/../../../config" && pwd) + +# A server-side dry run of a valid policy goes through the webhook and stores +# nothing, so it succeeds only once the webhook serves. +PROBE='apiVersion: galactic.datumapis.com/v1alpha1 +kind: TrafficPolicy +metadata: + name: webhook-probe + namespace: galactic-system +spec: + rules: + - name: probe + action: Deny' + +for site in dfw sjc iad; do + node=$(control_plane "${site}") + echo "Installing traffic policy CRDs and galactic-controller on ${site}..." + docker exec "${node}" rm -rf /galactic/traffic-policy + docker exec "${node}" mkdir -p /galactic/traffic-policy + docker cp "${CONFIG}/crd" "${node}:/galactic/traffic-policy/crd" + docker cp "${CONFIG}/galactic-controller" "${node}:/galactic/traffic-policy/galactic-controller" + apply_k "${node}" /galactic/traffic-policy/crd/ + docker exec "${node}" kubectl wait --for=condition=Established --timeout=60s \ + crd/trafficpolicies.galactic.datumapis.com crd/trafficpolicybindings.galactic.datumapis.com + # ServiceAccount and RBAC only; the Deployment comes from the lab overlay. + apply_k "${node}" /galactic/traffic-policy/galactic-controller/ + + # Same nesting as deploy-cni.sh: copy_to replaces the overlay, then the + # production base lands in its base/ leaf. + copy_to "${node}" galactic-controller + docker cp "${CONFIG}/galactic-controller/base" "${node}:/galactic/resources/galactic-controller/base" + apply_k "${node}" /galactic/resources/galactic-controller/ + docker exec "${node}" kubectl -n galactic-system rollout status deployment/galactic-controller --timeout=180s + + echo "Waiting for ${site}'s webhook to admit a policy..." + for i in $(seq 1 60); do + if printf '%s\n' "${PROBE}" | docker exec -i "${node}" kubectl create --dry-run=server -f - >/dev/null 2>&1; then + echo "${site}: webhook serving" + break + fi + if [ "${i}" -eq 60 ]; then + echo "error: ${site}'s galactic-controller webhook did not admit a policy within 120s" >&2 + printf '%s\n' "${PROBE}" | docker exec -i "${node}" kubectl create --dry-run=server -f - >&2 || true + exit 1 + fi + sleep 2 + done +done + +echo "Done." diff --git a/deploy/containerlab/scripts/publish-attachments.sh b/deploy/containerlab/scripts/publish-attachments.sh new file mode 100755 index 00000000..e57d7115 --- /dev/null +++ b/deploy/containerlab/scripts/publish-attachments.sh @@ -0,0 +1,89 @@ +#!/bin/bash +# publish-attachments.sh — make a tenant's VPCAttachments and write their +# status, the job the attachment controller does in production. +# +# Every galactic NetworkAttachmentDefinition in the namespace is one +# attachment, named after the NAD. One the tenant's own manifests do not +# define (only ns60's do, for the labels its NetworkRules select by) is +# created from the NAD's vpc. Each one's status then gets what its pod +# reports once scheduled: +# +# - vpc, vpcAttachment: the NAD's identifiers; +# - node, podName: where the pod using the NAD (as its +# v1.multus-cni.io/default-network) runs; +# - hostInterface: the host side of its veth, GH +# (internal/plumbing/intf), which galactic-router's traffic policy +# reconciler binds the attachment's policies to. +# +# ns60's NetworkRules wait on node to select a backend; every tenant +# interface waits on hostInterface, since the lab enforces traffic policy and +# an interface with no published bindings denies everything. +# +# Usage: publish-attachments.sh [site...] +set -euo pipefail + +if [ "$#" -lt 2 ]; then + echo "usage: $(basename "$0") [site...]" >&2 + exit 1 +fi + +NS="$1" +shift + +SCRIPT_DIR=$(cd "$(dirname "$0")" && pwd) +source "${SCRIPT_DIR}/lib.sh" + +for site in "$@"; do + cp=$(control_plane "${site}") + for nad in $(docker exec "${cp}" kubectl get network-attachment-definitions -n "${NS}" \ + -o jsonpath='{.items[*].metadata.name}'); do + config=$(docker exec "${cp}" kubectl get network-attachment-definitions -n "${NS}" "${nad}" \ + -o jsonpath='{.spec.config}') + vpc=$(jq -r '[.plugins[] | select(.type == "galactic-veth" or .type == "galactic-tap")][0].vpc // empty' <<<"${config}") + attachment=$(jq -r '[.plugins[] | select(.type == "galactic-veth" or .type == "galactic-tap")][0].vpcattachment // empty' <<<"${config}") + if [ -z "${vpc}" ] || [ -z "${attachment}" ]; then + continue # not a galactic network + fi + + if ! docker exec "${cp}" kubectl get vpcattachment -n "${NS}" "${nad}" >/dev/null 2>&1; then + docker exec -i "${cp}" kubectl apply -f - >/dev/null <&2 + exit 1 + fi + sleep 2 + done + node=$(docker exec "${cp}" kubectl get pod -n "${NS}" "${pod}" -o jsonpath='{.spec.nodeName}') + host=$(printf 'G%09s%03sH' "${vpc}" "${attachment}" | tr ' ' '0') + + docker exec "${cp}" kubectl patch vpcattachment -n "${NS}" "${nad}" --subresource=status --type=merge \ + -p "{\"status\":{\"vpc\":\"${vpc}\",\"vpcAttachment\":\"${attachment}\",\"node\":\"${node}\",\"podName\":\"${pod}\",\"hostInterface\":\"${host}\"}}" >/dev/null + echo "${site}: ${NS}/${nad} -> ${node} (${pod}, ${host})" + done +done diff --git a/deploy/containerlab/scripts/publish-ns60-attachments.sh b/deploy/containerlab/scripts/publish-ns60-attachments.sh deleted file mode 100755 index 6a67af04..00000000 --- a/deploy/containerlab/scripts/publish-ns60-attachments.sh +++ /dev/null @@ -1,40 +0,0 @@ -#!/bin/bash -# publish-ns60-attachments.sh — write the status of ns60's VPCAttachments, -# the job the attachment controller does in production. -# -# Each site's NetworkRules select ns60's backends through these attachments -# (resources/tenants/ns60//vpcattachments.yaml). An attachment is a -# backend only once its status names its VPC and its node, so this waits for -# each attachment's pod (same "instance" label) to be scheduled and copies its -# node into the attachment's status. -# -# Usage: publish-ns60-attachments.sh [site...] -set -euo pipefail - -if [ "$#" -lt 1 ]; then - echo "usage: $(basename "$0") [site...]" >&2 - exit 1 -fi - -SCRIPT_DIR=$(cd "$(dirname "$0")" && pwd) -source "${SCRIPT_DIR}/lib.sh" - -NS=ns60 -VPC=60 - -for site in "$@"; do - cp=$(control_plane "${site}") - for attachment in $(docker exec "${cp}" kubectl get vpcattachments -n "${NS}" -l app=backend \ - -o jsonpath='{.items[*].metadata.name}'); do - instance=$(docker exec "${cp}" kubectl get vpcattachment -n "${NS}" "${attachment}" \ - -o jsonpath='{.metadata.labels.instance}') - docker exec "${cp}" kubectl wait -n "${NS}" --for=condition=PodScheduled pod \ - -l "app=backend,instance=${instance}" --timeout=120s >/dev/null - pod=$(docker exec "${cp}" kubectl get pods -n "${NS}" -l "app=backend,instance=${instance}" \ - -o jsonpath='{.items[0].metadata.name}') - node=$(docker exec "${cp}" kubectl get pod -n "${NS}" "${pod}" -o jsonpath='{.spec.nodeName}') - docker exec "${cp}" kubectl patch vpcattachment -n "${NS}" "${attachment}" --subresource=status --type=merge \ - -p "{\"status\":{\"vpc\":\"${VPC}\",\"vpcAttachment\":\"${VPC}${instance}\",\"node\":\"${node}\",\"podName\":\"${pod}\"}}" >/dev/null - echo "${site}: ${NS}/${attachment} -> ${node} (${pod})" - done -done diff --git a/deploy/containerlab/scripts/verify-gateway-ingress.sh b/deploy/containerlab/scripts/verify-gateway-ingress.sh index ac7129dd..79d6c1fd 100755 --- a/deploy/containerlab/scripts/verify-gateway-ingress.sh +++ b/deploy/containerlab/scripts/verify-gateway-ingress.sh @@ -158,6 +158,8 @@ checked=0 for site in "${SITES[@]}"; do cp=$(control_plane "${site}") vip=${VIP[${site}]} + # --node checks one gateway, so a site without it is not checked at all. + [ -z "${only}" ] || [[ " ${GATEWAYS[${site}]} " == *" ${only} "* ]] || continue backends=$(pod_names "${cp}" ns60 app=backend) if [ -z "${backends}" ]; then fail "${site}: no ns60 backend pod (run deploy:ns60)" diff --git a/deploy/containerlab/scripts/verify-nat-active-active.sh b/deploy/containerlab/scripts/verify-nat-active-active.sh index 28b3b48b..9d48a275 100755 --- a/deploy/containerlab/scripts/verify-nat-active-active.sh +++ b/deploy/containerlab/scripts/verify-nat-active-active.sh @@ -131,9 +131,12 @@ members() { echo "galactic_cni_egress_pool_members{class=\"$1\",state=\"$2\"}"; # --- tenants -------------------------------------------------------------------- -# Tenant pods on the compute node, as ns/pod. +# Tenant pods on the compute node, as ns/pod. ns60's VIP backends are left +# out: the lab enforces traffic policy, and their egress grants only the +# PublicService path (resources/tenants/policies/public-service/), so they +# cannot egress through a shard at all. mapfile -t TENANTS < <(kc get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name} {.spec.nodeName}{"\n"}{end}' | - awk -v n="${COMPUTE}" '$2 == n && $1 ~ /^ns[0-9]+\// {print $1}') + awk -v n="${COMPUTE}" '$2 == n && $1 ~ /^ns[0-9]+\// && $1 !~ /^ns60\// {print $1}') [ "${#TENANTS[@]}" -ge 3 ] || die "need at least 3 tenant pods on ${COMPUTE}, found ${#TENANTS[@]}" FRESH_POD=$(kc -n ns10 get pods -o jsonpath='{.items[0].metadata.name}') FRESH_POD="ns10/${FRESH_POD}" diff --git a/deploy/containerlab/scripts/verify-traffic-policy.sh b/deploy/containerlab/scripts/verify-traffic-policy.sh new file mode 100755 index 00000000..a34754b1 --- /dev/null +++ b/deploy/containerlab/scripts/verify-traffic-policy.sh @@ -0,0 +1,272 @@ +#!/bin/bash +# verify-traffic-policy.sh — prove the lab's interface traffic policy is +# installed, forwards what each tenant's bindings grant and drops what they do +# not, on every forwarding path, and counts each decision where it was made. +# +# 1. Every TrafficPolicyBinding reports both directions Applied on its node. +# 2. Private: an ns20 TCP connection from dfw to sjc reaches a listener on +# 8080, and never reaches the same listener on 9, which sjc's ingress +# policy denies (rule deny-discard) and counts. +# 3. PublicNAT: ns10 on dfw reaches remote-host through a NAT shard; ns60's +# backend-a on dfw, whose egress grants only PublicService, does not, and +# dfw counts the no_match denial on its egress. +# 4. PublicService: with a temporary deny-udp rule ahead of ns60's +# allow-service on dfw, the dfw VIP's UDP check fails and the rule counts +# it; with the fixture restored, it passes again. +# 5. Built-in default: unbinding ns30's private-b ingress makes it deny +# everything as default-deny-all, counted under the built-in identity; +# rebinding restores it. +# +# Steps 4 and 5 change live policy; an exit trap restores the fixtures. +set -euo pipefail + +SCRIPT_DIR=$(cd "$(dirname "$0")" && pwd) +source "${SCRIPT_DIR}/lib.sh" + +METRICS_PORT=9179 +REMOTE_HOST="2001:db8:1:40::2" +BUILTIN_UID="builtin:default-deny-all:v1" +DFW=$(control_plane dfw) +SJC=$(control_plane sjc) +rc=0 + +fail() { + echo "FAIL $*" >&2 + rc=1 +} + +# metric_sum WORKER NAME LABEL=VALUE... — the sum of every NAME sample on +# WORKER's galactic-router whose labels include each LABEL="VALUE". +metric_sum() { + local worker="$1" name="$2" + shift 2 + local lines + lines=$(docker exec "${worker}" curl -fsS "http://localhost:${METRICS_PORT}/metrics" | grep "^${name}{" || true) + for kv in "$@"; do + lines=$(grep -F "${kv%%=*}=\"${kv#*=}\"" <<<"${lines}" || true) + done + awk '{s += $NF} END {printf "%d\n", s}' <<<"${lines}" +} + +# wait_applied SITE NS BINDING — until BINDING's status shows both +# directions Applied on some node, at the binding's current generation. +wait_applied() { + local cp + cp=$(control_plane "$1") + for _ in $(seq 1 60); do + if docker exec "${cp}" kubectl get trafficpolicybinding -n "$2" "$3" -o json | jq -e ' + .metadata.generation as $g + | [.status.nodes[]? + | select(.ingress.state == "Applied" and .egress.state == "Applied" + and .ingress.observedGeneration == $g)] | length > 0' >/dev/null; then + return 0 + fi + sleep 2 + done + return 1 +} + +# wait_policy_applied SITE NS POLICY — until every binding naming POLICY +# reports its current generation installed for that direction. +wait_policy_applied() { + local cp gen + cp=$(control_plane "$1") + gen=$(docker exec "${cp}" kubectl get trafficpolicy -n "$2" "$3" -o jsonpath='{.metadata.generation}') + for _ in $(seq 1 60); do + if docker exec "${cp}" kubectl get trafficpolicybindings -n "$2" -o json | jq -e \ + --arg p "$3" --argjson g "${gen}" ' + [.items[].status.nodes[]? | (.ingress, .egress) + | select(.policy.name == $p)] + | length > 0 and all(.state == "Applied" and .policy.generation == $g)' >/dev/null; then + return 0 + fi + sleep 2 + done + return 1 +} + +# tcp_reaches CP NS POD ADDRESS PORT — whether POD's TCP connection to +# [ADDRESS]:PORT is accepted within 5 seconds. +tcp_reaches() { + docker exec "$1" kubectl exec -n "$2" "$3" -- \ + timeout 6 sh -c "echo probe | socat -T 5 -u - TCP6:[$4]:$5,connect-timeout=5" >/dev/null 2>&1 +} + +# listen CP NS POD PORT — a one-shot TCP listener on PORT in POD, for 20s. +listen() { + docker exec "$1" kubectl exec -n "$2" "$3" -- \ + timeout 20 socat -u "TCP6-LISTEN:$4,reuseaddr" OPEN:/dev/null >/dev/null 2>&1 & + sleep 1 +} + +out=$(mktemp) +# restore re-applies the two fixtures steps 4 and 5 change and waits for the +# router to install them, so a check run after this one sees the lab as +# deployed. +restore() { + set +e + apply_k "${DFW}" /galactic/resources/tenants/ns60/policy/ >/dev/null + apply_k "${DFW}" /galactic/resources/tenants/ns30/policy/ >/dev/null + wait_policy_applied dfw ns60 lab-ingress || echo "warning: ns60's lab-ingress not reinstalled on dfw" >&2 + wait_applied dfw ns30 private-b || echo "warning: ns30/private-b not rebound on dfw" >&2 + rm -f "${out}" +} +trap restore EXIT + +echo "--- 1. every binding is installed on its node" +for site in dfw sjc iad; do + cp=$(control_plane "${site}") + while read -r ns name; do + [ -z "${ns}" ] && continue + if wait_applied "${site}" "${ns}" "${name}"; then + echo "ok ${site}: ${ns}/${name} applied in both directions" + else + fail "${site}: ${ns}/${name} is not applied in both directions:" + docker exec "${cp}" kubectl get trafficpolicybinding -n "${ns}" "${name}" -o jsonpath='{.status}' >&2 + echo >&2 + fi + done < <(docker exec "${cp}" kubectl get trafficpolicybindings -A \ + -o jsonpath='{range .items[*]}{.metadata.namespace} {.metadata.name}{"\n"}{end}') +done + +echo "--- 2. Private: a granted port connects, a denied one does not" +src=$(pod_name "${DFW}" ns20) +dst=$(pod_name "${SJC}" ns20) +dst_ip=$(pod_ip6 "${SJC}" ns20 "${dst}") +listen "${SJC}" ns20 "${dst}" 8080 +if tcp_reaches "${DFW}" ns20 "${src}" "${dst_ip}" 8080; then + echo "ok ns20 dfw -> sjc TCP 8080 connects" +else + fail "ns20 dfw -> sjc TCP 8080 did not connect" +fi +before=$(metric_sum sjc-worker galactic_cni_traffic_policy_rule_hits_total \ + namespace=ns20 direction=ingress rule=deny-discard) +listen "${SJC}" ns20 "${dst}" 9 +if tcp_reaches "${DFW}" ns20 "${src}" "${dst_ip}" 9; then + fail "ns20 dfw -> sjc TCP 9 connected through a deny rule" +else + echo "ok ns20 dfw -> sjc TCP 9 is dropped" +fi +after=$(metric_sum sjc-worker galactic_cni_traffic_policy_rule_hits_total \ + namespace=ns20 direction=ingress rule=deny-discard) +if [ "${after}" -gt "${before}" ]; then + echo "ok sjc counted the denial on ns20's ingress: deny-discard ${before} -> ${after}" +else + fail "sjc's deny-discard counter for ns20 did not move (${before} -> ${after})" +fi + +echo "--- 3. PublicNAT: granted egress reaches remote-host, ungranted does not" +ns10=$(pod_name "${DFW}" ns10) +if tcp_reaches "${DFW}" ns10 "${ns10}" "${REMOTE_HOST}" 80; then + echo "ok ns10 dfw -> remote-host TCP 80 connects through a NAT shard" +else + fail "ns10 dfw -> remote-host TCP 80 did not connect" +fi +backend=$(pod_name "${DFW}" ns60 app=backend,instance=a) +before=$(metric_sum dfw-worker galactic_cni_traffic_policy_fallback_packets_total \ + namespace=ns60 attachment=backend-a direction=egress reason=no_match) +if tcp_reaches "${DFW}" ns60 "${backend}" "${REMOTE_HOST}" 80; then + fail "ns60 backend-a -> remote-host connected though its egress grants only PublicService" +else + echo "ok ns60 backend-a -> remote-host is dropped" +fi +after=$(metric_sum dfw-worker galactic_cni_traffic_policy_fallback_packets_total \ + namespace=ns60 attachment=backend-a direction=egress reason=no_match) +if [ "${after}" -gt "${before}" ]; then + echo "ok dfw counted backend-a's egress as no_match: ${before} -> ${after}" +else + fail "dfw's no_match counter for backend-a's egress did not move (${before} -> ${after})" +fi + +echo "--- 4. PublicService: a deny rule stops the VIP's UDP, and lifting it restores it" +docker exec -i "${DFW}" kubectl apply -f - >/dev/null <<'EOF' +apiVersion: galactic.datumapis.com/v1alpha1 +kind: TrafficPolicy +metadata: + name: lab-ingress + namespace: ns60 +spec: + rules: + - name: deny-udp + action: Deny + match: + path: PublicService + protocol: UDP + - name: allow-service + action: Allow + match: + path: PublicService +EOF +if ! wait_policy_applied dfw ns60 lab-ingress; then + fail "dfw did not install ns60's edited lab-ingress" +fi +before=$(metric_sum dfw-worker galactic_cni_traffic_policy_rule_hits_total \ + namespace=ns60 direction=ingress rule=deny-udp) +if "${SCRIPT_DIR}/verify-gateway-ingress.sh" --node dfw-worker2 >"${out}" 2>&1; then + fail "dfw's VIP passed its TCP and UDP check with UDP denied to its backends" +elif grep -q 'udp from port .*: no reply' "${out}" && ! grep -q 'tcp from port' "${out}"; then + echo "ok dfw's VIP still serves TCP, and its UDP draws no reply while denied" +else + fail "dfw's VIP check failed, but not only on UDP:" + sed 's/^/ /' "${out}" >&2 +fi +after=$(metric_sum dfw-worker galactic_cni_traffic_policy_rule_hits_total \ + namespace=ns60 direction=ingress rule=deny-udp) +if [ "${after}" -gt "${before}" ]; then + echo "ok dfw counted the VIP's UDP under deny-udp: ${before} -> ${after}" +else + fail "dfw's deny-udp counter for ns60 did not move (${before} -> ${after})" +fi +apply_k "${DFW}" /galactic/resources/tenants/ns60/policy/ >/dev/null +if ! wait_policy_applied dfw ns60 lab-ingress; then + fail "dfw did not reinstall ns60's lab-ingress" +fi +if "${SCRIPT_DIR}/verify-gateway-ingress.sh" --node dfw-worker2 >"${out}" 2>&1; then + echo "ok dfw's VIP serves TCP and UDP again" +else + fail "dfw's VIP still fails with the fixture restored:" + sed 's/^/ /' "${out}" >&2 +fi + +echo "--- 5. Built-in default: an unbound direction denies, rebinding restores it" +a=$(pod_name "${DFW}" ns30) +b=$(pod_name "${DFW}" ns30 app=private-b) +b_ip=$(pod_ip6 "${DFW}" ns30 "${b}") +docker exec "${DFW}" kubectl patch trafficpolicybinding -n ns30 private-b --type=json \ + -p '[{"op":"remove","path":"/spec/ingressPolicyRef"}]' >/dev/null +for _ in $(seq 1 60); do + got=$(docker exec "${DFW}" kubectl get trafficpolicybinding -n ns30 private-b \ + -o jsonpath='{.status.nodes[0].ingress.policy.uid}') + [ "${got}" = "${BUILTIN_UID}" ] && break + sleep 2 +done +if [ "${got}" != "${BUILTIN_UID}" ]; then + fail "ns30/private-b's ingress did not fall back to the built-in default (uid ${got:-none})" +fi +before=$(metric_sum dfw-worker galactic_cni_traffic_policy_rule_hits_total \ + namespace=ns30 attachment=private-b direction=ingress policy_uid="${BUILTIN_UID}") +if docker exec "${DFW}" kubectl exec -n ns30 "${a}" -- ping -6 -c 3 -W 2 "${b_ip}" >/dev/null 2>&1; then + fail "ns30 private -> private-b answered with private-b on the built-in default" +else + echo "ok ns30 private -> private-b is dropped on the built-in default" +fi +after=$(metric_sum dfw-worker galactic_cni_traffic_policy_rule_hits_total \ + namespace=ns30 attachment=private-b direction=ingress policy_uid="${BUILTIN_UID}") +if [ "${after}" -gt "${before}" ]; then + echo "ok dfw counted the default deny-all as a rule hit: ${before} -> ${after}" +else + fail "dfw's default deny-all counter for ns30/private-b did not move (${before} -> ${after})" +fi +apply_k "${DFW}" /galactic/resources/tenants/ns30/policy/ >/dev/null +if wait_applied dfw ns30 private-b && + docker exec "${DFW}" kubectl exec -n ns30 "${a}" -- ping -6 -c 3 -W 2 "${b_ip}" >/dev/null 2>&1; then + echo "ok ns30 private -> private-b answers again once rebound" +else + fail "ns30 private -> private-b does not answer after rebinding" +fi + +if [ "${rc}" -ne 0 ]; then + echo "traffic policy: FAILED" >&2 + exit 1 +fi +echo "traffic policy: every path forwards what it grants and drops, and counts, what it does not" diff --git a/docs/README.md b/docs/README.md index fed97eba..8383dfea 100644 --- a/docs/README.md +++ b/docs/README.md @@ -60,6 +60,9 @@ data flow, module reference, known constraints): - [traffic-policy/controller.md](traffic-policy/controller.md) — `galactic-controller --role=cell`: the TrafficPolicy webhook, the reserved default policy, binding resolution status, flags and install order. +- [traffic-policy/rollout.md](traffic-policy/rollout.md) — turning traffic + policy on in a cell: install order, writing grants, auditing with the + counters before enforcing, and backing out. - [traffic-policy/node.md](traffic-policy/node.md) — galactic-router's node side: enabling it, what each binding applies, per-node status, the `galactic_cni_traffic_policy_*` metrics and the `policy explain` and diff --git a/docs/agents/ARCHITECTURE-GATEWAY.md b/docs/agents/ARCHITECTURE-GATEWAY.md index 5d6bfc38..f6820d7e 100644 --- a/docs/agents/ARCHITECTURE-GATEWAY.md +++ b/docs/agents/ARCHITECTURE-GATEWAY.md @@ -904,7 +904,7 @@ that tag into `config/galactic-gateway/base`. - **`XDP_REDIRECT` into a veth needs NAPI enabled on the *peer*, which is a lab-only concern.** `edgedsr.c` returns `XDP_REDIRECT` whenever the route to the backend egresses an interface other than the ingress one. In native mode that calls the egress device's `ndo_xdp_xmit`, and veth's implementation silently discards the frame unless the peer end has NAPI enabled — which for a veth means the peer runs its own XDP program or has GRO turned on. A containerlab peer inside an FRR/transit container has neither by default, so a redirect there can fail exactly the way this datapath's original bug did: no drop counter moves, because `bpf_redirect()` itself succeeded and the discard happens later in `xdp_do_redirect`. Both programs are affected, and `edge_return` more so: it attaches to compute-facing links, which are veth pairs in the lab and may be veth or a plain NIC in production. `bpftool prog tracelog` and the `xdp:xdp_redirect_err` tracepoint are the diagnostics; `ethtool -K gro on` is the lab fix. A production uplink (physical NIC, or a bond slave per the bullet above) implements `ndo_xdp_xmit` natively and has no peer to ask about. This is the redirect-side analogue of the `XDP_TX`-on-veth observability quirk noted above. - **The return path is opt-in per node and fails closed at startup, not silently.** `edge_return` attaches only where `GALACTIC_GATEWAY_INTERNAL_INTERFACES` names an interface; a node that needs it and does not set it drops every reply in `KUBE-FORWARD` with nothing to say why, exactly as before this program existed. Where it *is* set, an attach failure is fatal to `setupGatewayDatapath` and the pod crash-loops rather than running with the forward half working and replies dying — the same all-or-nothing choice `edgeattach.Attach` already makes across a bond's slaves. - **A backend in another site replies through that site's edge node.** The gateway can pick a backend anywhere its `NetworkRule` reaches, and the reply then leaves through the edge node in front of *that* backend, whose own `vip_addr_table` must hold the same anycast VIP or the reply meets the unmodified kernel path and dies. Within one cluster the reconciler guarantees this; across clusters it depends on the same `NetworkRule` existing on both sides, which nothing in this repo enforces. -- **Backends depend on an attachment controller outside this repo.** `selectRuleBackends` skips a `VPCAttachment` whose `status.vpc` is not the rule's VPC, and keeps a selected one pending until its `status.node` is set. Galactic writes neither field; controllers in datum-cloud/cloud do, `status.node` from the attachment's `BGPAdvertisement`. Without them an attachment with no `status.vpc` is silently unselected, one with `status.vpc` but no `status.node` is pending with `status.node not set by datum-cloud/cloud`, and a rule with no other backend never loads; the containerlab lab patches the status by hand (`deploy/containerlab/scripts/publish-ns60-attachments.sh`). See #821. +- **Backends depend on an attachment controller outside this repo.** `selectRuleBackends` skips a `VPCAttachment` whose `status.vpc` is not the rule's VPC, and keeps a selected one pending until its `status.node` is set. Galactic writes neither field; controllers in datum-cloud/cloud do, `status.node` from the attachment's `BGPAdvertisement`. Without them an attachment with no `status.vpc` is silently unselected, one with `status.vpc` but no `status.node` is pending with `status.node not set by datum-cloud/cloud`, and a rule with no other backend never loads; the containerlab lab patches the status by hand (`deploy/containerlab/scripts/publish-attachments.sh`). See #821. - **IPv4 VIPs are accepted but not served.** A rule that predates the CRD's IPv6-only rule can still hold an IPv4 VIP, but `edge_lb` and `edge_return` parse only IPv6, `edgemap` refuses IPv4 keys, and backend nodes translate only IPv6 VIPs. So `buildDesiredRule` leaves IPv4 VIPs, plain or IPv4-mapped, out of the engine, no `-v4` advertisement is made, and each gateway node reports `/IPv4Served=False` (`Unsupported`) on the rule. The address never draws traffic that nothing answers, and the rule's IPv6 VIP keeps serving. #705 tracks serving IPv4 VIPs. - **`vip_table` has no active GC beyond crash-recovery reconcile.** By design (see Key Design Decisions above) — DSR keeps no flow state to leak in the first place. - **Egress is out of this binary's scope, not unimplemented.** Egress translation is a separate, sharded tier (`galactic-nat`, its own binary — see `cmd/galactic-nat` and `internal/controller/egressshard_controller.go`). `NetworkRule`/this datapath remain ingress-only: external client → VIP → tenant backend. diff --git a/docs/enhancements/networking/interface-traffic-policy/README.md b/docs/enhancements/networking/interface-traffic-policy/README.md index 873d8c35..e18df01a 100644 --- a/docs/enhancements/networking/interface-traffic-policy/README.md +++ b/docs/enhancements/networking/interface-traffic-policy/README.md @@ -777,7 +777,9 @@ enforcement to 6.8, which the pull requests call out. Default deny is a deliberate behavior change. The enforcement mode in `tp_state` is `off`, `audit` or `enforce`, set per node by -`GALACTIC_CNI_TRAFFIC_POLICY` and defaulting to `off` until the final change. +`GALACTIC_CNI_TRAFFIC_POLICY`. It defaults to `off`; only the containerlab lab +enforces by default. [docs/traffic-policy/rollout.md](../../../traffic-policy/rollout.md) +is the operator's step-by-step guide. - `off` changes nothing. Maps and programs are loaded, but `tp_capture` and `tp_decide` return immediately. diff --git a/docs/gateway/configuration.md b/docs/gateway/configuration.md index ef3f5f4c..f1a4b579 100644 --- a/docs/gateway/configuration.md +++ b/docs/gateway/configuration.md @@ -522,7 +522,7 @@ attachment whose `status.vpc` is set but whose `status.node` is not stays pending with `status.node not set by datum-cloud/cloud`. A rule whose attachments are all pending or unselected does not load. The containerlab lab runs none of these controllers, so -`deploy/containerlab/scripts/publish-ns60-attachments.sh` patches +`deploy/containerlab/scripts/publish-attachments.sh` patches `status.vpc` and `status.node` in by hand. Every accepted, non-deleting `NetworkRule` in the namespace is served by diff --git a/docs/traffic-policy/rollout.md b/docs/traffic-policy/rollout.md new file mode 100644 index 00000000..1567f307 --- /dev/null +++ b/docs/traffic-policy/rollout.md @@ -0,0 +1,136 @@ +# Rolling out interface traffic policy + +Interface traffic policy denies by default: once a node enforces it, a tenant +interface forwards only what its bindings grant, in each direction. That is a +deliberate behavior change, so turning it on in a cell is a migration. This +guide is the order to do it in, what to check at each step, and how to back +out. The design is in +[the traffic policy proposal](../enhancements/networking/interface-traffic-policy/README.md); +[controller.md](controller.md) and [node.md](node.md) are the component +references. + +TrafficPolicy is not Kubernetes NetworkPolicy and is unrelated to BGPPolicy: +it filters a VPC workload's own interface, by forwarding path, address, +protocol and port, and neither reads nor replaces either of those. + +## The three modes + +`GALACTIC_CNI_TRAFFIC_POLICY` on `galactic-cni` sets each node's mode: + +| Mode | Decides | Counts | Drops | Use | +| --------- | ------- | ------ | ----- | ------------------------------------------ | +| `off` | no | no | no | The default; the programs are loaded inert | +| `audit` | yes | yes | no | Measuring what enforcement would deny | +| `enforce` | yes | yes | yes | The steady state | + +`audit` is for measuring a migration, not a compatibility mode: it forwards +everything, so a cell left in it enforces nothing. There is no legacy allow, +and nothing generates reverse rules: a workload that answers connections needs +an ingress grant for the requests and an egress grant for the replies. + +In `audit` and `enforce`, a new interface's host side stays down until every +enforcement program is attached and verified, and CNI ADD fails on a node +whose kernel failed the preflight (Linux 6.8 or later, `HASH_OF_MAPS` and +`bpf_for_each_map_elem`). Production and staging run Talos v1.13.10, on +kernel 6.18. + +## Before you start + +Check every node of the cell: + +- runs a galactic release that includes traffic policy, for `galactic-cni`, + `galactic-router` and `galactic-controller`; +- has the `galactic.datumapis.com` CRDs (`config/crd/`), cert-manager, and the + cloud operator's VPCAttachment CRD installed; +- publishes `status.node` and `status.hostInterface` on every VPCAttachment, + which the cloud operator writes. An attachment without them is never bound + on any node, and denies everything once enforced. + +## Steps + +1. **Install the cell controller.** Apply `config/galactic-controller/` (its + ServiceAccount and RBAC), then `config/galactic-controller/base/`. Its + webhook fails closed, so wait until it admits a policy before anyone + writes one: a server-side dry run of a valid TrafficPolicy succeeds. It + provisions `default-deny-all` in every namespace that holds an attachment. + +2. **Turn on the node reconciler.** Set `GALACTIC_ROUTER_TRAFFIC_POLICY=true` + on the plain `galactic-router` DaemonSet, never the route reflector. With + the mode still `off`, nothing changes for any packet, but every binding + now reports what each node would install in `status.nodes`, and + `galactic_cni_traffic_policy_*` appears on each router's metrics port. + +3. **Write the grants.** For every VPCAttachment, create a + TrafficPolicyBinding of the same name with an ingress and an egress + reference, and the policies they name. A direction without a reference + gets the built-in `default-deny-all`, which denies everything. Grant by + path (`Private`, `PublicNAT`, `PublicService`) as well as by address and + port, so a rule for one path never opens another. Check that every + binding's `status.nodes` shows both directions `Applied`: + + ```sh + kubectl get trafficpolicybindings -A -o json | jq -r ' + .items[] | . as $b | .status.nodes[]? + | select(.ingress.state != "Applied" or .egress.state != "Applied") + | "\($b.metadata.namespace)/\($b.metadata.name) on \(.node): \(.ingress.state)/\(.egress.state)"' + ``` + +4. **Recreate workloads that predate the release.** An interface created + before the release gets its policy state from `galactic-cni` at startup, + but not its addresses, so its ARP and its neighbor discovery from a + global address count as ordinary traffic rather than link operations, and + are denied unless a rule grants them. Recreate those workloads; a new + interface records its addresses at ADD. + +5. **Audit.** Set `GALACTIC_CNI_TRAFFIC_POLICY=audit` on `galactic-cni`, one + node or one site at a time. Every decision is now counted, and nothing is + dropped. Leave it long enough to see each workload's normal traffic, + including periodic jobs, then find what enforcement would deny: + + ```promql + # Would be denied by no matching rule, or by a missing binding. + sum by (namespace, attachment, direction, reason) ( + increase(galactic_cni_traffic_policy_fallback_packets_total[1h]) + ) > 0 + + # Would be denied by a rule, including the built-in default. + sum by (namespace, attachment, direction, policy, rule) ( + increase(galactic_cni_traffic_policy_rule_hits_total{action="deny"}[1h]) + ) > 0 + + # Interfaces still on the built-in default. + count by (namespace, attachment, direction) ( + galactic_cni_traffic_policy_rule_hits_total{policy_uid="builtin:default-deny-all:v1"} + ) + ``` + + `policy_unavailable` means an interface has no installed policy at all: + its attachment is not bound on its node yet, or its binding's reference is + missing. `classification_error` means a packet's path could not be + determined, typically because the node's NAT shard locators have not + synced. Fix the grants and repeat until each query is empty, or shows only + traffic you mean to deny. `galactic-router policy explain` decides a + packet against a policy offline, to check a rule before relying on it. + +6. **Enforce.** Set `GALACTIC_CNI_TRAFFIC_POLICY=enforce`, again a node or a + site at a time, and watch the same queries: they now count drops. + Changing the mode takes effect at once on every existing interface and + needs no workload restart. + +## Backing out + +Set `GALACTIC_CNI_TRAFFIC_POLICY=off`. The datapath stops deciding at once and +forwards as before; bindings, status and counters stay, so moving forward +again resumes where it stopped. Turning the reconciler off +(`GALACTIC_ROUTER_TRAFFIC_POLICY=false`) leaves the installed snapshots in +place but stops updating them, so do that only with the mode `off`. + +## In the containerlab lab + +The lab in `deploy/containerlab/` enforces from the start: its `galactic-cni` +sets `enforce`, its routers run the reconciler, and every tenant fixture +binds both directions to a policy set granting only the paths its checks use +(`resources/tenants/policies/`). `task verify:traffic-policy` checks that +every binding is installed, and that each path forwards what it grants and +drops, and counts, what it does not. It is the place to try a policy change +before writing it for a cell.