diff --git a/AGENTS.md b/AGENTS.md index a96e037d..d81c793a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -96,7 +96,7 @@ See [docs/node-labels.md](docs/node-labels.md) for the full strategy — why `ga - **`config/galactic-router/overlays/control/`** — the BGP route-reflector role (`galactic-router-rr`, `GALACTIC_ROUTER_REFLECTOR=true`); opt-in only, requires `galactic.datumapis.com/galactic: control` (stays at zero replicas otherwise). Mutually exclusive with `router`'s `galactic: router` value on the same node — one label key, single-valued, so a route-reflector node structurally can't also run plain-mode `galactic-router`. `GALACTIC_ROUTER_BGP_LOCAL_ADDRESS` is auto-detected from the host's `lo` interface by default; see the comments in `daemonset-patch.yaml` for when to override it. - Apply both roles with `kubectl apply -k config/galactic-router/`, or a single role with e.g. `kubectl apply -k config/galactic-router/overlays/router/`. `galactic-router` no longer runs bundled inside `galactic-gateway`'s pod — see `config/galactic-gateway/` below. - **`config/galactic-gateway/`** — the edge XDP DSR load-balancer control plane, a single-container `galactic-gateway` DaemonSet — `galactic-router` used to run as a second container in this same pod; it's now a fully independent DaemonSet (via `galactic.datumapis.com/galactic: router`, which a gateway node may carry like any other), so a crash on either side no longer takes the other's *pod* down with it, not just the other's binary. `config/galactic-gateway/{serviceaccount.yaml,rbac.yaml}` (safe/idempotent to apply cluster-wide) are what `kubectl apply -k config/galactic-gateway/` applies; `config/galactic-gateway/base/` (requiring nodes labeled both `galactic.datumapis.com/node: edge` and `galactic.datumapis.com/gateway: enabled`, so it runs on edge nodes alongside `galactic-vrf`) is **not** included in that kustomization and is **not** applied as-is — the same exemption as `config/fabric-router/` below, for the same reason: `GALACTIC_GATEWAY_PUBLIC_INTERFACE` differs per gateway node and has no generic default. `GALACTIC_GATEWAY_SRV6_ADDRESS` needs no per-node value: left unset, the gateway derives it at startup from its node's `BGPRouter` locator and node ID, and a set value overrides that (see `internal/config/gateway.go`'s `EnvGatewaySRv6Address` doc comment). It's designed to be instantiated once per gateway node by a further overlay that pins it to one node (`kubernetes.io/hostname`) and sets that node's own public interface; see `deploy/containerlab/resources/galactic-gateway/` for a worked example (one overlay per lab edge node). Also **not** part of the root `config/kustomization.yaml`'s default resource list, matching `config/fabric-router/`'s exemption. -- **`config/galactic-nat/`** — the sharded egress translation (NAT66/NAT64) control plane, one `galactic-nat` DaemonSet per cluster on every node labeled both `galactic.datumapis.com/node: edge` and `galactic.datumapis.com/nat: enabled`, alongside `galactic-vrf`. It is independent of `galactic-gateway`. A shard's identity (SID, masquerade addresses, NAT64 prefix) comes from the `EgressShard` targeting its node, not from the DaemonSet. The base leaves `GALACTIC_NAT_XDP_ATTACH` at its default, `direct`. On a node that also runs the gateway, set it to `dispatch`: an interface takes one native XDP program, so the shard and the gateway (which defaults to `GALACTIC_GATEWAY_XDP_ATTACH=dispatch`) each run from their own slot of the node's shared, pinned XDP dispatcher (`internal/plumbing/ebpf/xdpdispatch`), and either can restart or be turned off (`GALACTIC_*_DATAPATH_ENABLED=false`) without touching the other's traffic. `hack/xdp-dispatch-release.sh` detaches the dispatcher from a node. Compute nodes reach their own site's shards over SRv6 via `GALACTIC_CNI_EGRESS_SHARD_SIDS`. `kubectl apply -k config/galactic-nat/` applies only the ServiceAccount/RBAC; `base/` is excluded because uplink auto-detection is deployment-specific (see `config/galactic-nat/kustomization.yaml`). Not part of the root `config/kustomization.yaml`. +- **`config/galactic-nat/`** — the sharded egress translation (NAT66/NAT64) control plane, one `galactic-nat` DaemonSet per cluster on every node labeled both `galactic.datumapis.com/node: edge` and `galactic.datumapis.com/nat: enabled`, alongside `galactic-vrf`. It is independent of `galactic-gateway`. A shard's identity (SID, masquerade addresses, NAT64 prefix) comes from the `EgressShard` targeting its node, not from the DaemonSet. The base leaves `GALACTIC_NAT_XDP_ATTACH` at its default, `direct`. On a node that also runs the gateway, set it to `dispatch`: an interface takes one native XDP program, so the shard and the gateway (which defaults to `GALACTIC_GATEWAY_XDP_ATTACH=dispatch`) each run from their own slot of the node's shared, pinned XDP dispatcher (`internal/plumbing/ebpf/xdpdispatch`), and either can restart or be turned off (`GALACTIC_*_DATAPATH_ENABLED=false`) without touching the other's traffic. `hack/xdp-dispatch-release.sh` detaches the dispatcher from a node. Compute nodes reach their own site's shards over SRv6: by default each VRF uses the first reachable shard in `GALACTIC_CNI_EGRESS_SHARD_SIDS`; with `GALACTIC_CNI_EGRESS_MODE=hashed` on `galactic-cni`, `usid_egress` spreads tenant addresses across the cluster's `EgressShard`s through one immutable Maglev snapshot per translation class (NAT66, and each NAT64 prefix, holding only the shards that can translate it) and per-address pins, and `spec.drain` takes a shard out of rotation without breaking its pinned sessions; delete a shard only after draining it (see [docs/nat/configuration.md](docs/nat/configuration.md#active-active-egress) and the rollout checklist [docs/nat/active-active-rollout.md](docs/nat/active-active-rollout.md)). `kubectl apply -k config/galactic-nat/` applies only the ServiceAccount/RBAC; `base/` is excluded because uplink auto-detection is deployment-specific (see `config/galactic-nat/kustomization.yaml`). Not part of the root `config/kustomization.yaml`. - **`config/fabric-router/`** — the FRR underlay eBGP DaemonSet (`fabric-router`; `galactic-router` needs a working underlay before it can start). Unlike `config/galactic-router/`, this is a single flat DaemonSet with no `default`/`rr`-style role split — its affinity matches any node labeled `galactic.datumapis.com/fabric: router` (a dedicated mode-enum label, independent of whatever `galactic.datumapis.com/node` role value or `galactic.datumapis.com/galactic` mode that node also carries) directly, since there's no env/config difference between running on a regular node vs. those roles — fabric-router itself is identical everywhere; only the per-node BGP underlay config it reads from the ConfigMap differs. That affinity can legitimately match more than one node per cluster, and BGP underlay config (hostname, router-id, interface addresses, remote-AS) inherently differs per physical node — so each node's config lives in its own `fabric-router.` ConfigMap (a `frr.conf` key, plus optional `daemons`/`vtysh.conf` overrides of the image defaults) rather than one shared ConfigMap for the whole DaemonSet. `fabric-config-agent` (`cmd/fabric-config-agent`, `internal/fabricconfig`, built into the `fabric-router` image) reads it from the API: as the `frr-init` init container it validates (`vtysh -C`) and installs it, holding the pod in `Init` until a valid one exists; as the `config-agent` sidecar it applies later changes live with `frr-reload.py`, so an edit to one node's ConfigMap neither restarts the pod nor touches other nodes, and reports `Applied`/`Rejected`/`ReloadFailed`/`RestartRequired` events on the ConfigMap. The sidecar also hard-resets any IPv6 session announcing an IPv4-mapped next hop (`internal/fabricconfig/nexthop.go`) and serves that check's metrics on the node's port `9343`. A third container, `frr-exporter` (upstream `tynany/frr_exporter`, pinned by digest), reads the daemons' vty sockets in the same pod-scoped `/run/frr` and serves BGP peer state, message counters and prefix counts for both underlay families on the node's port `9342` (`hostNetwork`); `task verify:fabric-metrics` in `deploy/containerlab/` checks it on every lab node. The image's default `daemons` loads bgpd's BMP module (`-M bmp`), so a node's `frr.conf` can stream to a BMP collector (`config/bmp-collector/`) with a `bmp targets` block; loading it alone streams nothing, and a change to the modules needs a pod restart rather than a reload. Every lab node's `frr.conf` streams its pre-policy Adj-RIB-In and Loc-RIB for both families, sourced from `lo` (`source-interface lo`), since the underlay carries loopbacks but not link subnets. Production `frr.conf`s come from infra's renderer, which has to emit the same block for production nodes to stream. A node with no ConfigMap of its own falls back, with a `DeprecatedConfigSource` warning, to key `frr.conf.` of the legacy shared `fabric-config` ConfigMap. **Not** part of the root `config/kustomization.yaml` and not covered by `kubectl apply -k config/` — unlike every other component here, it has no generic default: the deployer must hand-author one `fabric-router.` ConfigMap per matching node before applying `kubectl apply -k config/fabric-router/` (which also applies the agent's ServiceAccount and namespaced Role). The opt-in Component `config/fabric-router/components/fabric-api` adds the fabric-api sidecar to this DaemonSet's pod (see `config/fabric-api/` below); applying it, and every later fabric-api image change, recreates the pod and restarts FRR with it, so roll it out as a routing change. - **`config/bmp-collector/`** — a single-replica GoBMP Deployment and its `bmp-collector` Service (TCP `5000`), the BMP (RFC 7854) collector `galactic-router` and `fabric-router` stream to; `galactic-router` does so when `GALACTIC_ROUTER_BMP_STATIONS` names it (`bmp-collector.galactic-system.svc:5000`; see [docs/router/configuration.md](docs/router/configuration.md)). It parses every Peer Up/Down, route-monitoring and statistics message to JSON on stdout, so the cluster's log pipeline is its store; an overlay swapping `--dump=console` for `--kafka-server` publishes to Kafka instead. Opt-in, like `config/fabric-router/`: not part of the root `config/kustomization.yaml`, since nothing streams to it until a router is configured to. The containerlab lab runs one on `iad-worker3`'s host network (`deploy/containerlab/resources/bmp-collector/iad/`), because its three clusters cannot reach each other's Services, and points every node's `fabric-router` and the EVPN route reflector at it. Of the `galactic-router`s, only the reflector streams: its pre-policy Adj-RIB-In already holds every node's advertisements. - **`config/fabric-api/`** — the fabric looking glass's cell-wide prerequisites and per-cell gateway. `kubectl apply -k config/fabric-api/` applies only the `fabric-api-gateway` and `fabric-api-janitor` ServiceAccounts and RBAC (the janitor's delete access is deliberately a separate identity from the gateway's, which can only read `FabricQuery` and write its status); safe and idempotent in any cell. `config/fabric-api/base/` (the 2-replica leader-elected gateway Deployment, its ClusterIP debug Service, a PodDisruptionBudget and the `*/15` janitor CronJob) is **not** included in that kustomization and is **not** applied as-is, for the same reason as `config/galactic-gateway/base/`: it needs per-cell values — a `fabric-api` ConfigMap (`site`, `cell`, `cluster-name`) and the cell name in its certificate's URI SAN — supplied by a per-cell overlay; see `deploy/containerlab/resources/fabric-api/gateway/` for a worked example. `config/fabric-api/crd/` is a generated `FabricQuery` CRD for the lab and tests only; infra owns the CRD (and the per-cell `fabric-api` cert-manager `Issuer`) in every cell. The node sidecar is the `config/fabric-router/components/fabric-api` Component, which also adds a `fabric-api-certs` DaemonSet (`fabric-api certsync`) that owns the cert-manager csi-driver volume and copies the certificate to the node's `/run/fabric-api/tls` for the sidecar: a csi volume blocks its pod until the certificate is issued, so keeping it out of the `fabric-router` pod means a broken Issuer stalls only the certs pod, never FRR. The overlay's `CELL_NAME` replacement applies to that DaemonSet's `fabric-api-csi` volume. Not part of the root `config/kustomization.yaml`: fabric-api is opt-in. See [docs/fabric-api/configuration.md](docs/fabric-api/configuration.md) and [docs/fabric-api/api.md](docs/fabric-api/api.md). diff --git a/config/monitoring/prometheusrule.yaml b/config/monitoring/prometheusrule.yaml index 3dca910f..05cb5937 100644 --- a/config/monitoring/prometheusrule.yaml +++ b/config/monitoring/prometheusrule.yaml @@ -355,6 +355,161 @@ spec: Prometheus cannot reach galactic-nat's metrics port on {{ $labels.node }} (9182), so GalacticNatSessionTableEvictingLive, GalacticNatPortExhausted and GalacticNatUplinkRxQueueStalled cannot fire for it. + # From galactic-cni (job galactic-cni, port 9180): its egress shard groups, + # one per translation class (nat66, or nat64:), and its own + # control loop. A node in ordered mode with no group left publishes no + # pool series, so these rules key off galactic_cni_egress_mode and + # galactic_cni_egress_group_routes, which galactic-cni always exports, + # rather than read missing pool series as a healthy pool. + - name: galactic-cni-egress + rules: + # A class's group has shards, or routes still name it, but no shard + # takes tenants: every candidate is unreachable from this node or + # draining, or the cluster has none left. Every tenant of that class + # on the node without a live pin has no egress. + - alert: GalacticEgressPoolEmpty + expr: | + sum by (node, class) (galactic_cni_egress_pool_members{job="galactic-cni", state="active"}) == 0 + and on (node, class) + ( + sum by (node, class) (galactic_cni_egress_pool_members{job="galactic-cni", state=~"draining|unreachable"}) > 0 + or + sum by (node, class) (galactic_cni_egress_group_routes{job="galactic-cni"}) > 0 + ) + for: 2m + labels: + severity: critical + service: galactic + team: connect + annotations: + summary: "No egress shard takes {{ $labels.class }} traffic from {{ $labels.node }}" + runbook_url: "https://github.com/datum-cloud/infra/blob/main/docs/runbooks/galactic/README.md#galacticegresspoolempty" + description: >- + No egress shard in {{ $labels.node }}'s {{ $labels.class }} group has taken new tenants for 2 + minutes while the group has shards or routes, so new {{ $labels.class }} egress from that node is + dropped. galactic_cni_egress_pool_members shows whether its shards are unreachable (no route to their + SID over an SRv6 uplink) or draining. + # Fewer shards take tenants than the operator said the class should + # have. A class with no eligible shard at all is + # GalacticEgressClassUnserved instead. + - alert: GalacticEgressPoolBelowMinimum + expr: | + sum by (node, class) (galactic_cni_egress_pool_members{job="galactic-cni", state="active"}) + < on (node) group_left max by (node) (galactic_cni_egress_pool_min_active{job="galactic-cni"}) + and on (node, class) + sum by (node, class) (galactic_cni_egress_pool_members{job="galactic-cni", state=~"active|draining|unreachable"}) > 0 + and on (node, class) + max by (node, class) (galactic_cni_egress_pool_enabled{job="galactic-cni"}) == 1 + for: 10m + labels: + severity: warning + service: galactic + team: connect + annotations: + summary: "{{ $labels.node }} has fewer active {{ $labels.class }} egress shards than the pool minimum" + runbook_url: "https://github.com/datum-cloud/infra/blob/main/docs/runbooks/galactic/README.md#galacticegresspoolbelowminimum" + description: >- + {{ $labels.node }} has had {{ $value }} active {{ $labels.class }} egress shards for 10 + minutes, below the GALACTIC_CNI_EGRESS_POOL_MIN_ACTIVE the cluster is configured for. The remaining + shards carry the metro's whole egress load for that class. galactic_cni_egress_pool_members shows + which shards are draining or unreachable. + # Every EgressShard in the cluster is ready but none can translate + # this class: none has a masquerade address in its family, or none + # translates this NAT64 prefix. galactic-cni then installs no route + # for the class, so its traffic has no egress. + - alert: GalacticEgressClassUnserved + expr: | + sum by (node, class) (galactic_cni_egress_pool_members{job="galactic-cni", state=~"active|draining|unreachable"}) == 0 + and on (node, class) + sum by (node, class) (galactic_cni_egress_pool_members{job="galactic-cni", state="ineligible"}) > 0 + for: 15m + labels: + severity: warning + service: galactic + team: connect + annotations: + summary: "No egress shard can translate {{ $labels.class }} for {{ $labels.node }}" + runbook_url: "https://github.com/datum-cloud/infra/blob/main/docs/runbooks/galactic/README.md#galacticegressclassunserved" + description: >- + {{ $labels.node }} is configured for {{ $labels.class }} egress, but none of the cluster's + EgressShards can translate it: none has a masquerade address in that family, or none translates that + NAT64 prefix. galactic-cni logs the shards it kept out. Either give a shard that capability or remove + the prefix from GALACTIC_CNI_NAT64_PREFIX. + # usid_egress dropped tenant packets bound for a shard group: the + # group was empty or disabled, its Maglev table named a shard that + # was not alive, or the packet could not be parsed. Drops in every + # window for 10 minutes are not a passing rebuild. + - alert: GalacticEgressShardGroupDropping + expr: >- + sum by (node, result) ( + increase(galactic_cni_egress_shard_selections_total{job="galactic-cni", result=~"group_empty|shard_dead|parse_failed"}[5m]) + ) > 0 + for: 10m + labels: + severity: warning + service: galactic + team: connect + annotations: + summary: "{{ $labels.node }} is dropping tenant egress bound for a shard group ({{ $labels.result }})" + runbook_url: "https://github.com/datum-cloud/infra/blob/main/docs/runbooks/galactic/README.md#galacticegressshardgroupdropping" + description: >- + usid_egress on {{ $labels.node }} has dropped tenant packets bound for an egress shard group + ({{ $labels.result }}) in every 5-minute window for 10 minutes, about {{ $value | humanize }} in the + last one. group_empty means no shard takes tenants; shard_dead means the group was not rebuilt + after a shard became unreachable, which points at galactic-cni's egress sweep. + # Hashed mode is configured but galactic-cni has never listed the + # cluster's EgressShards: RBAC, the API server or the CRD is missing. + # Its groups keep whatever they last held, and a new node has none. + - alert: GalacticEgressShardWatchNotSynced + expr: | + max by (node) (galactic_cni_egress_mode{job="galactic-cni", mode="hashed"}) == 1 + and on (node) + max by (node) (galactic_cni_egress_shard_watch_synced{job="galactic-cni"}) == 0 + for: 10m + labels: + severity: warning + service: galactic + team: connect + annotations: + summary: "galactic-cni on {{ $labels.node }} cannot list EgressShards" + runbook_url: "https://github.com/datum-cloud/infra/blob/main/docs/runbooks/galactic/README.md#galacticegressshardwatchnotsynced" + description: >- + galactic-cni on {{ $labels.node }} runs hashed egress mode but its EgressShard watch has not listed + the cluster for 10 minutes. Its shard groups keep their last members, and a node that never + listed them has no group at all. Check galactic-cni's logs, its get/list/watch permission on + egressshards, and that the EgressShard CRD is installed. + # A sweep step keeps failing: publishing a group, reading or rewriting + # egress_route_table, listing EgressShards, or retiring groups. While + # it fails, membership and next hops go stale, and a move back to + # ordered mode does not finish. + - alert: GalacticEgressSweepFailing + expr: | + sum by (node, stage) (increase(galactic_cni_egress_sweep_errors_total{job="galactic-cni"}[10m])) > 0 + for: 15m + labels: + severity: warning + service: galactic + team: connect + annotations: + summary: "galactic-cni's egress sweep on {{ $labels.node }} keeps failing at {{ $labels.stage }}" + runbook_url: "https://github.com/datum-cloud/infra/blob/main/docs/runbooks/galactic/README.md#galacticegresssweepfailing" + description: >- + galactic-cni on {{ $labels.node }} has failed its egress sweep at the {{ $labels.stage }} step in + every 10-minute window for 15 minutes. Shard group membership and next hops are going stale, and a + switch back to ordered mode leaves routes on the groups until it succeeds. + - alert: GalacticCNIMetricsDown + expr: up{job="galactic-cni"} == 0 + for: 5m + labels: + severity: warning + service: galactic + team: connect + annotations: + summary: "galactic-cni metrics on {{ $labels.node }} cannot be scraped" + runbook_url: "https://github.com/datum-cloud/infra/blob/main/docs/runbooks/galactic/README.md#galacticcnimetricsdown" + description: >- + Prometheus cannot reach galactic-cni's metrics port on {{ $labels.node }} (9180), so no egress + shard group alert can fire for it. # fabric-api: the looking glass's cell gateway and node sidecars. None of # these fire merely because nobody ran a query: each is either a state # the components report continuously (availability, credentials, diff --git a/config/monitoring/tests/rules-test.yaml b/config/monitoring/tests/rules-test.yaml index 27d26677..85f177ca 100644 --- a/config/monitoring/tests/rules-test.yaml +++ b/config/monitoring/tests/rules-test.yaml @@ -674,6 +674,248 @@ tests: GalacticNatSessionTableEvictingLive, GalacticNatPortExhausted and GalacticNatUplinkRxQueueStalled cannot fire for it. + # Hashed egress, NAT66 class: the node's only shard becomes unreachable at + # 5m. GalacticEgressPoolEmpty fires after 2m, BelowMinimum after 10m. The + # NAT64 class on another node is retired (enabled 0) with no member and no + # route left, and fires nothing. + - interval: 30s + input_series: + - series: 'galactic_cni_egress_pool_enabled{job="galactic-cni",node="dfw-worker",class="nat66"}' + values: "1x40" + - series: 'galactic_cni_egress_pool_min_active{job="galactic-cni",node="dfw-worker"}' + values: "1x40" + - series: 'galactic_cni_egress_pool_members{job="galactic-cni",node="dfw-worker",class="nat66",state="active"}' + values: "1x10 0x30" + - series: 'galactic_cni_egress_pool_members{job="galactic-cni",node="dfw-worker",class="nat66",state="unreachable"}' + values: "0x10 1x30" + - series: 'galactic_cni_egress_pool_members{job="galactic-cni",node="dfw-worker",class="nat66",state="draining"}' + values: "0x40" + - series: 'galactic_cni_egress_pool_enabled{job="galactic-cni",node="iad-worker",class="nat64:2001:db8:64::/96"}' + values: "0x40" + - series: 'galactic_cni_egress_pool_min_active{job="galactic-cni",node="iad-worker"}' + values: "1x40" + - series: 'galactic_cni_egress_pool_members{job="galactic-cni",node="iad-worker",class="nat64:2001:db8:64::/96",state="active"}' + values: "0x40" + - series: 'galactic_cni_egress_pool_members{job="galactic-cni",node="iad-worker",class="nat64:2001:db8:64::/96",state="unreachable"}' + values: "0x40" + - series: 'galactic_cni_egress_group_routes{job="galactic-cni",node="iad-worker",class="nat64:2001:db8:64::/96"}' + values: "0x40" + alert_rule_test: + - eval_time: 6m + alertname: GalacticEgressPoolEmpty + exp_alerts: [] + - eval_time: 8m + alertname: GalacticEgressPoolEmpty + exp_alerts: + - exp_labels: + severity: critical + service: galactic + team: connect + node: dfw-worker + class: nat66 + exp_annotations: + summary: "No egress shard takes nat66 traffic from dfw-worker" + runbook_url: "https://github.com/datum-cloud/infra/blob/main/docs/runbooks/galactic/README.md#galacticegresspoolempty" + description: >- + No egress shard in dfw-worker's nat66 group has taken new tenants for 2 + minutes while the group has shards or routes, so new nat66 egress from that node is + dropped. galactic_cni_egress_pool_members shows whether its shards are unreachable (no route to their + SID over an SRv6 uplink) or draining. + - eval_time: 14m + alertname: GalacticEgressPoolBelowMinimum + exp_alerts: [] + - eval_time: 16m + alertname: GalacticEgressPoolBelowMinimum + exp_alerts: + - exp_labels: + severity: warning + service: galactic + team: connect + node: dfw-worker + class: nat66 + exp_annotations: + summary: "dfw-worker has fewer active nat66 egress shards than the pool minimum" + runbook_url: "https://github.com/datum-cloud/infra/blob/main/docs/runbooks/galactic/README.md#galacticegresspoolbelowminimum" + description: >- + dfw-worker has had 0 active nat66 egress shards for 10 + minutes, below the GALACTIC_CNI_EGRESS_POOL_MIN_ACTIVE the cluster is configured for. The remaining + shards carry the metro's whole egress load for that class. galactic_cni_egress_pool_members shows + which shards are draining or unreachable. + + # A group with no candidate left but routes still naming it (every + # EgressShard deleted) fires PoolEmpty: the missing members must not read + # as an idle pool. + - interval: 30s + input_series: + - series: 'galactic_cni_egress_pool_members{job="galactic-cni",node="sjc-worker",class="nat66",state="active"}' + values: "0x20" + - series: 'galactic_cni_egress_pool_members{job="galactic-cni",node="sjc-worker",class="nat66",state="unreachable"}' + values: "0x20" + - series: 'galactic_cni_egress_group_routes{job="galactic-cni",node="sjc-worker",class="nat66"}' + values: "4x20" + alert_rule_test: + - eval_time: 3m + alertname: GalacticEgressPoolEmpty + exp_alerts: + - exp_labels: + severity: critical + service: galactic + team: connect + node: sjc-worker + class: nat66 + exp_annotations: + summary: "No egress shard takes nat66 traffic from sjc-worker" + runbook_url: "https://github.com/datum-cloud/infra/blob/main/docs/runbooks/galactic/README.md#galacticegresspoolempty" + description: >- + No egress shard in sjc-worker's nat66 group has taken new tenants for 2 + minutes while the group has shards or routes, so new nat66 egress from that node is + dropped. galactic_cni_egress_pool_members shows whether its shards are unreachable (no route to their + SID over an SRv6 uplink) or draining. + + # A NAT64 class no shard can translate fires ClassUnserved after 15m, and + # neither PoolEmpty nor BelowMinimum, which would both be the wrong story. + - interval: 30s + input_series: + - series: 'galactic_cni_egress_pool_enabled{job="galactic-cni",node="dfw-worker",class="nat64:2001:db8:64::/96"}' + values: "1x40" + - series: 'galactic_cni_egress_pool_min_active{job="galactic-cni",node="dfw-worker"}' + values: "1x40" + - series: 'galactic_cni_egress_pool_members{job="galactic-cni",node="dfw-worker",class="nat64:2001:db8:64::/96",state="active"}' + values: "0x40" + - series: 'galactic_cni_egress_pool_members{job="galactic-cni",node="dfw-worker",class="nat64:2001:db8:64::/96",state="ineligible"}' + values: "2x40" + alert_rule_test: + - eval_time: 16m + alertname: GalacticEgressClassUnserved + exp_alerts: + - exp_labels: + severity: warning + service: galactic + team: connect + node: dfw-worker + class: nat64:2001:db8:64::/96 + exp_annotations: + summary: "No egress shard can translate nat64:2001:db8:64::/96 for dfw-worker" + runbook_url: "https://github.com/datum-cloud/infra/blob/main/docs/runbooks/galactic/README.md#galacticegressclassunserved" + description: >- + dfw-worker is configured for nat64:2001:db8:64::/96 egress, but none of the cluster's + EgressShards can translate it: none has a masquerade address in that family, or none translates that + NAT64 prefix. galactic-cni logs the shards it kept out. Either give a shard that capability or remove + the prefix from GALACTIC_CNI_NAT64_PREFIX. + - eval_time: 16m + alertname: GalacticEgressPoolEmpty + exp_alerts: [] + - eval_time: 16m + alertname: GalacticEgressPoolBelowMinimum + exp_alerts: [] + + # Steady group_empty drops fire after 10m; a single burst of shard_dead + # drops, the race between a withdrawal and the rebuild, does not. + - interval: 30s + input_series: + - series: 'galactic_cni_egress_shard_selections_total{job="galactic-cni",node="dfw-worker",result="group_empty"}' + values: "0+4x40" + - series: 'galactic_cni_egress_shard_selections_total{job="galactic-cni",node="sjc-worker",result="shard_dead"}' + values: "0x5 20x35" + alert_rule_test: + - eval_time: 9m + alertname: GalacticEgressShardGroupDropping + exp_alerts: [] + - eval_time: 16m + alertname: GalacticEgressShardGroupDropping + exp_alerts: + - exp_labels: + severity: warning + service: galactic + team: connect + node: dfw-worker + result: group_empty + exp_annotations: + summary: "dfw-worker is dropping tenant egress bound for a shard group (group_empty)" + runbook_url: "https://github.com/datum-cloud/infra/blob/main/docs/runbooks/galactic/README.md#galacticegressshardgroupdropping" + description: >- + usid_egress on dfw-worker has dropped tenant packets bound for an egress shard group + (group_empty) in every 5-minute window for 10 minutes, about 40 in the + last one. group_empty means no shard takes tenants; shard_dead means the group was not rebuilt + after a shard became unreachable, which points at galactic-cni's egress sweep. + + # Hashed mode with a watch that never syncs fires after 10m; ordered mode + # with the same unsynced watch does not. + - interval: 30s + input_series: + - series: 'galactic_cni_egress_mode{job="galactic-cni",node="dfw-worker",mode="hashed"}' + values: "1x30" + - series: 'galactic_cni_egress_shard_watch_synced{job="galactic-cni",node="dfw-worker"}' + values: "0x30" + - series: 'galactic_cni_egress_mode{job="galactic-cni",node="iad-worker",mode="hashed"}' + values: "0x30" + - series: 'galactic_cni_egress_shard_watch_synced{job="galactic-cni",node="iad-worker"}' + values: "0x30" + alert_rule_test: + - eval_time: 11m + alertname: GalacticEgressShardWatchNotSynced + exp_alerts: + - exp_labels: + severity: warning + service: galactic + team: connect + node: dfw-worker + exp_annotations: + summary: "galactic-cni on dfw-worker cannot list EgressShards" + runbook_url: "https://github.com/datum-cloud/infra/blob/main/docs/runbooks/galactic/README.md#galacticegressshardwatchnotsynced" + description: >- + galactic-cni on dfw-worker runs hashed egress mode but its EgressShard watch has not listed + the cluster for 10 minutes. Its shard groups keep their last members, and a node that never + listed them has no group at all. Check galactic-cni's logs, its get/list/watch permission on + egressshards, and that the EgressShard CRD is installed. + + # Refresh failing in every window fires after 15m; a single failure does + # not. + - interval: 30s + input_series: + - series: 'galactic_cni_egress_sweep_errors_total{job="galactic-cni",node="dfw-worker",stage="refresh"}' + values: "0+1x60" + - series: 'galactic_cni_egress_sweep_errors_total{job="galactic-cni",node="sjc-worker",stage="apply"}' + values: "0x5 1x55" + alert_rule_test: + - eval_time: 26m + alertname: GalacticEgressSweepFailing + exp_alerts: + - exp_labels: + severity: warning + service: galactic + team: connect + node: dfw-worker + stage: refresh + exp_annotations: + summary: "galactic-cni's egress sweep on dfw-worker keeps failing at refresh" + runbook_url: "https://github.com/datum-cloud/infra/blob/main/docs/runbooks/galactic/README.md#galacticegresssweepfailing" + description: >- + galactic-cni on dfw-worker has failed its egress sweep at the refresh step in + every 10-minute window for 15 minutes. Shard group membership and next hops are going stale, and a + switch back to ordered mode leaves routes on the groups until it succeeds. + + - interval: 30s + input_series: + - series: 'up{job="galactic-cni",node="dfw-worker"}' + values: "0x20" + alert_rule_test: + - eval_time: 6m + alertname: GalacticCNIMetricsDown + exp_alerts: + - exp_labels: + severity: warning + service: galactic + team: connect + job: galactic-cni + node: dfw-worker + exp_annotations: + summary: "galactic-cni metrics on dfw-worker cannot be scraped" + runbook_url: "https://github.com/datum-cloud/infra/blob/main/docs/runbooks/galactic/README.md#galacticcnimetricsdown" + description: >- + Prometheus cannot reach galactic-cni's metrics port on dfw-worker (9180), so no egress + shard group alert can fire for it. + # fabric-api: a gateway with executions in flight and no completions for # 10m fires; one that keeps completing queries, and one that is idle, do # not. diff --git a/deploy/containerlab/README.md b/deploy/containerlab/README.md index 68192f94..aa3c0453 100644 --- a/deploy/containerlab/README.md +++ b/deploy/containerlab/README.md @@ -477,58 +477,59 @@ task deploy ## Tasks -| Task | Description | -|---------------------------|-------------------------------------------------------------------------------------------------------------| -| `build` | Build all container images (node, galactic-router, galactic-cni, frr, host) | -| `build:node` | Build the custom `kindest/node:galactic` image | -| `build:galactic-router` | Build the galactic-router container from Go source | -| `build:galactic-cni` | Build the galactic-cni installer image | -| `build:galactic-gateway` | Build the galactic-gateway edge load balancer image | -| `build:frr` | Build the FRR container from Alpine edge | -| `build:remote-host` | Build the off-fabric nginx host image | -| `deploy` | Build images, apply host sysctls, and deploy the lab | -| `deploy:topology` | Deploy the ContainerLab topology (transit routers) | -| `deploy:clusters` | Create the three Kind clusters and export their kubeconfigs | -| `deploy:images` | Load container images into Kind clusters | -| `deploy:system` | Install BGP and VPC CRDs; apply the galactic-system namespace and shared RBAC¹ | -| `deploy:cni` | Install Cilium and Multus, then the galactic-cni DaemonSet | -| `deploy:fabric` | Apply FRR DaemonSets to all clusters | -| `deploy:bmp-collector` | Install the BMP collector on iad-worker3, where the route reflector streams to it | -| `deploy:galactic-router` | Apply galactic-router DaemonSets and BGP CRs | -| `deploy:galactic-gateway` | Install galactic-gateway on every edge node, with each site's VIP rules (after `deploy:scenarios`) | -| `deploy:scenarios` | Deploy all VPC test scenarios | -| `deploy:ns10` | Deploy ns10 test VPC (IPv6-only, fd20 ULA) | -| `deploy:ns20` | Deploy ns20 test VPC (dual-stack, fd20 ULA + IPv4) | -| `deploy:ns30` | Deploy ns30 test VPC (dfw only, 2 pods) | -| `deploy:ns40` | Deploy ns40 test VPC (iad only, 2 pods) | -| `deploy:ns60` | Deploy ns60 test VPC (two VIP backends per site, on one node) | -| `verify:fabric-metrics` | Scrape every fabric-router's frr-exporter; fail on a failed collector or a session not Established | -| `verify:bmp` | Check the route reflector and every fabric-router stream to the BMP collector, and that it agrees with each on every session's state | -| `verify:underlay` | Ping every underlay loopback from tr1 over both IPv4 and IPv6 | -| `verify:gateway` | Check every NetworkGateway is Ready, every NetworkRule Accepted, every generated ServiceVIPBinding Bound | -| `verify:gateway-ingress` | TCP and UDP from the off-fabric host to each site's VIP, through every gateway node in turn | -| `verify:gateway-restart` | Restart dfw-worker2's gateway and shard in turn while the other carries traffic (#710) | -| `verify:gateway-detach` | Turn dfw-worker2's gateway datapath off: the ingress check must fail, then pass (not in `verify`) | -| `verify:nat-datapath` | Full egress round trip to the off-fabric host, IPv6 (NAT66) and IPv4 (NAT64) | -| `verify:nat-return-route` | Prove a reply from outside the fabric reaches each shard's masquerade addresses | -| `verify:nat-local` | Prove each site's tenants egress through that site's own edge shard | -| `verify:nat-collision` | Prove two tenants sharing a ULA on one node do not share a translation | -| `verify:mss-clamp` | Prove full-size TCP segments cross the fabric, out through NAT and across sites | -| `verify:pmtu` | Prove a packet too big for the fabric draws an ICMP error from its own node, between sites and through NAT | -| `verify:nat-icmp` | Prove ping, traceroute and path MTU discovery work through each site's egress shard | -| `verify:scenarios` | Verify ping across all VPC test scenarios | -| `verify:ns10` | Verify ns10 ping (IPv6-only, 3-site mesh) | -| `verify:ns20` | Verify ns20 ping (dual-stack, 3-site mesh) | -| `verify:ns30` | Verify ns30 ping (dfw only, 2 pods) | -| `verify:ns40` | Verify ns40 ping (iad only, 2 pods) | -| `destroy` | Destroy the lab and remove all Kind clusters | -| `restart` | Full rebuild — destroy then redeploy | -| `rebuild` | Full rebuild — clean (destroy + delete images/artifacts) then redeploy | -| `inspect` | Show running nodes and management addresses | -| `graph` | Generate a draw.io diagram for the topology | -| `host-setup` | Apply required host sysctls (IPv6 forwarding, inotify limits) | -| `clean` | Destroy lab, delete built images, and remove lab artifacts | -| `test` | Run all verification checks | +| Task | Description | +| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | +| `build` | Build all container images (node, galactic-router, galactic-cni, frr, host) | +| `build:node` | Build the custom `kindest/node:galactic` image | +| `build:galactic-router` | Build the galactic-router container from Go source | +| `build:galactic-cni` | Build the galactic-cni installer image | +| `build:galactic-gateway` | Build the galactic-gateway edge load balancer image | +| `build:frr` | Build the FRR container from Alpine edge | +| `build:remote-host` | Build the off-fabric nginx host image | +| `deploy` | Build images, apply host sysctls, and deploy the lab | +| `deploy:topology` | Deploy the ContainerLab topology (transit routers) | +| `deploy:clusters` | Create the three Kind clusters and export their kubeconfigs | +| `deploy:images` | Load container images into Kind clusters | +| `deploy:system` | Install BGP and VPC CRDs; apply the galactic-system namespace and shared RBAC¹ | +| `deploy:cni` | Install Cilium and Multus, then the galactic-cni DaemonSet | +| `deploy:fabric` | Apply FRR DaemonSets to all clusters | +| `deploy:bmp-collector` | Install the BMP collector on iad-worker3, where the route reflector streams to it | +| `deploy:galactic-router` | Apply galactic-router DaemonSets and BGP CRs | +| `deploy:galactic-gateway` | Install galactic-gateway on every edge node, with each site's VIP rules (after `deploy:scenarios`) | +| `deploy:scenarios` | Deploy all VPC test scenarios | +| `deploy:ns10` | Deploy ns10 test VPC (IPv6-only, fd20 ULA) | +| `deploy:ns20` | Deploy ns20 test VPC (dual-stack, fd20 ULA + IPv4) | +| `deploy:ns30` | Deploy ns30 test VPC (dfw only, 2 pods) | +| `deploy:ns40` | Deploy ns40 test VPC (iad only, 2 pods) | +| `deploy:ns60` | Deploy ns60 test VPC (two VIP backends per site, on one node) | +| `verify:fabric-metrics` | Scrape every fabric-router's frr-exporter; fail on a failed collector or a session not Established | +| `verify:bmp` | Check the route reflector and every fabric-router stream to the BMP collector, and that it agrees with each on every session's state | +| `verify:underlay` | Ping every underlay loopback from tr1 over both IPv4 and IPv6 | +| `verify:gateway` | Check every NetworkGateway is Ready, every NetworkRule Accepted, every generated ServiceVIPBinding Bound | +| `verify:gateway-ingress` | TCP and UDP from the off-fabric host to each site's VIP, through every gateway node in turn | +| `verify:gateway-restart` | Restart dfw-worker2's gateway and shard in turn while the other carries traffic (#710) | +| `verify:gateway-detach` | Turn dfw-worker2's gateway datapath off: the ingress check must fail, then pass (not in `verify`) | +| `verify:nat-datapath` | Full egress round trip to the off-fabric host, IPv6 (NAT66) and IPv4 (NAT64) | +| `verify:nat-return-route` | Prove a reply from outside the fabric reaches each shard's masquerade addresses | +| `verify:nat-local` | Prove each site's tenants egress through that site's own edge shard | +| `verify:nat-collision` | Prove two tenants sharing a ULA on one node do not share a translation | +| `verify:mss-clamp` | Prove full-size TCP segments cross the fabric, out through NAT and across sites | +| `verify:pmtu` | Prove a packet too big for the fabric draws an ICMP error from its own node, between sites and through NAT | +| `verify:nat-icmp` | Prove ping, traceroute and path MTU discovery work through each site's egress shard | +| `verify:nat-active-active` | Hashed egress on dfw: both shards carry tenants, a failure and a drain move only theirs | +| `verify:scenarios` | Verify ping across all VPC test scenarios | +| `verify:ns10` | Verify ns10 ping (IPv6-only, 3-site mesh) | +| `verify:ns20` | Verify ns20 ping (dual-stack, 3-site mesh) | +| `verify:ns30` | Verify ns30 ping (dfw only, 2 pods) | +| `verify:ns40` | Verify ns40 ping (iad only, 2 pods) | +| `destroy` | Destroy the lab and remove all Kind clusters | +| `restart` | Full rebuild — destroy then redeploy | +| `rebuild` | Full rebuild — clean (destroy + delete images/artifacts) then redeploy | +| `inspect` | Show running nodes and management addresses | +| `graph` | Generate a draw.io diagram for the topology | +| `host-setup` | Apply required host sysctls (IPv6 forwarding, inotify limits) | +| `clean` | Destroy lab, delete built images, and remove lab artifacts | +| `test` | Run all verification checks | ¹ Network CRDs come from the latest commit on `datum-cloud/network`'s `main`, not the version `go.mod` requires. Set `NETWORK_REF=` to track a diff --git a/deploy/containerlab/Taskfile.yaml b/deploy/containerlab/Taskfile.yaml index 7bb85381..1081de51 100644 --- a/deploy/containerlab/Taskfile.yaml +++ b/deploy/containerlab/Taskfile.yaml @@ -398,6 +398,7 @@ tasks: - task: "verify:pmtu" - task: "verify:nat-icmp" - task: "verify:nat64-non-global" + - task: "verify:nat-active-active" - task: "verify:scenarios" # First, because every check after it rides these bonds: a member that @@ -567,6 +568,15 @@ tasks: cmds: - LAB={{.LAB}} ./scripts/verify-gateway-detach.sh + # Switches dfw's galactic-cni to hashed egress mode for its duration and + # back to ordered mode at the end, so the NAT checks around it, which expect + # dfw-worker2 to carry dfw's egress, see the lab as they left it. dfw is the + # only site with two shards. + "verify:nat-active-active": + desc: Verify dfw spreads egress across both shards in hashed mode, survives a shard failure and a drain, and returns to ordered mode + cmds: + - LAB={{.LAB}} ./scripts/verify-nat-active-active.sh + "verify:nat-sharding": desc: Verify every edge node's EgressShard is programmed and advertised cmds: diff --git a/deploy/containerlab/scripts/verify-nat-active-active.sh b/deploy/containerlab/scripts/verify-nat-active-active.sh new file mode 100755 index 00000000..5702c8d1 --- /dev/null +++ b/deploy/containerlab/scripts/verify-nat-active-active.sh @@ -0,0 +1,615 @@ +#!/bin/bash +# verify-nat-active-active.sh — acceptance check for hashed (active/active) +# egress on dfw, the lab's only site with two egress shards (dfw-worker2, +# dfw-worker3). The lab runs ordered mode; this switches dfw's galactic-cni to +# hashed mode for its duration and restores whatever it found at the end, +# whatever happens: the egress environment variables, both shards' drain +# flags, galactic-cni's ClusterRole, every route and address it adds. +# +# Every probe fails unless every request succeeds and every packet arrives at +# the off-fabric host from an expected masquerade address. A probe that sees +# no traffic fails; it never passes vacuously. +# +# 0. Preflight: both shards Ready and Programmed with distinct SIDs and +# masquerade addresses, each address routed back to its shard from the +# transit, the EgressShard CRD carrying spec.drain (read back after a +# write), galactic-cni allowed to watch EgressShards. A self-test then +# blackholes a fresh tenant address and checks the probe fails. +# 1. Flow hash, pins off: one tenant's TCP and UDP flows use both shards, +# over NAT66 and NAT64; ICMP works through both families. +# 2. Source hash, default pins: every flow from one tenant address, TCP, +# UDP and ICMP alike, leaves from one shard per family. +# 3. galactic-cni restart: a download in progress survives it, and every +# tenant keeps its shard (pins and slot generations survive). +# 4. Local failure [prohibit route]: one shard made unreachable from the +# compute node by a prohibit route over its locator, which is what the +# resolver sees when a route goes; this is not BGP. Its tenants move, +# new tenants go to the survivor, a download on the survivor completes. +# 5. BGP withdrawal [real BGP]: the shard node's FRR stops originating its +# SID; the compute node learns of it over BGP. Reports how long the +# group took to drop it, and to take it back after re-origination. +# 6. Drain: a download on the drained shard completes there, the tenant +# that owns it keeps its shard while it stays active, and fresh tenant +# addresses all go to the other shard, over NAT66 and NAT64. +# 7. All shards unavailable [prohibit routes]: fresh tenants get nothing, +# and the drops are counted as group_empty. +# 8. EgressShard watch failure: with its permission revoked and +# galactic-cni restarted, the watch never syncs, the metric says so, +# and the groups it published before keep forwarding. +# 9. Rollback: back to ordered mode, no route names a group, the groups +# are retired, and every tenant leaves from dfw-worker2, the first +# shard in dfw's list. +# +# SKIP_BGP=1 skips step 5, which waits on real BGP and can take minutes. +set -euo pipefail + +SCRIPT_DIR=$(cd "$(dirname "$0")" && pwd) +source "${SCRIPT_DIR}/lib.sh" + +CP=dfw-control-plane +COMPUTE=dfw-worker +REMOTE=clab-gvpc-remote-host +TRANSIT=clab-gvpc-tr4 +HOST6=2001:db8:1:40::2 +HOST4=11.1.40.2 +# HOST4 synthesized into the fabric's NAT64 prefix. +HOST4_SYNTH=2001:db8:64::b01:2802 +NAT64_CLASS='nat64:2001:db8:64::/96' +SHARDS=(dfw-worker2-egress dfw-worker3-egress) +SOAK_MB=12 +SOAK_BYTES=$((SOAK_MB * 1024 * 1024)) +CNI_METRICS=http://localhost:9180/metrics + +kc() { docker exec "${CP}" kubectl "$@"; } +log() { echo "$*"; } +die() { + echo "FAIL: $*" >&2 + exit 1 +} + +STATE=$(mktemp -d) + +# --- shard identities -------------------------------------------------------- + +declare -A V6 V4 SID NODE +for s in "${SHARDS[@]}"; do + V6[${s}]=$(kc -n galactic-system get egressshard "${s}" -o jsonpath='{.status.shardAddressIPv6}') + V4[${s}]=$(kc -n galactic-system get egressshard "${s}" -o jsonpath='{.status.shardAddressIPv4}') + SID[${s}]=$(kc -n galactic-system get egressshard "${s}" -o jsonpath='{.status.shardSID}') + NODE[${s}]=$(kc -n galactic-system get egressshard "${s}" -o jsonpath='{.spec.targetRef.name}') +done + +shard_of() { + local s + for s in "${SHARDS[@]}"; do + if [ "$1" = "${V6[${s}]}" ] || [ "$1" = "${V4[${s}]}" ]; then + echo "${s}" + return + fi + done + echo unknown +} + +# locator SHARD prints the /68 covering SHARD's SID and every tenant copy of it. +locator() { + python3 -c 'import ipaddress,sys; print(ipaddress.IPv6Network(sys.argv[1] + "/68", strict=False))' "${SID[$1]}" +} + +# sid64 SHARD prints the /64 the shard's node originates. +sid64() { + python3 -c 'import ipaddress,sys; print(ipaddress.IPv6Network(sys.argv[1] + "/64", strict=False))' "${SID[$1]}" +} + +# --- metrics ------------------------------------------------------------------ + +metric() { + docker exec "${COMPUTE}" curl -s --max-time 4 "${CNI_METRICS}" | awk -v m="$1" '$1 == m {print $2}' +} + +metric_sum() { + docker exec "${COMPUTE}" curl -s --max-time 4 "${CNI_METRICS}" | + awk -v p="$1" 'index($1, p) == 1 {s += $2} END {print s + 0}' +} + +wait_metric() { # NAME WANT TIMEOUT + local deadline=$((SECONDS + $3)) got="" + while [ ${SECONDS} -lt ${deadline} ]; do + got=$(metric "$1") + if [ "${got}" = "$2" ]; then return 0; fi + sleep 2 + done + die "$1 = ${got:-missing}, want $2 after $3s" +} + +members() { echo "galactic_cni_egress_pool_members{class=\"$1\",state=\"$2\"}"; } + +# --- tenants -------------------------------------------------------------------- + +# Tenant pods on the compute node, as ns/pod. +mapfile -t TENANTS < <(kc get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name} {.spec.nodeName}{"\n"}{end}' | + awk -v n="${COMPUTE}" '$2 == n && $1 ~ /^ns[0-9]+\// {print $1}') +[ "${#TENANTS[@]}" -ge 3 ] || die "need at least 3 tenant pods on ${COMPUTE}, found ${#TENANTS[@]}" +FRESH_POD=$(kc -n ns10 get pods -o jsonpath='{.items[0].metadata.name}') +FRESH_POD="ns10/${FRESH_POD}" + +pod_exec() { + local t="$1" + shift + kc -n "${t%%/*}" exec "${t#*/}" -- "$@" +} + +# pod_netns TENANT prints the network namespace name of TENANT on its node. +pod_netns() { + local sandbox + sandbox=$(docker exec "${COMPUTE}" crictl pods --name "${1#*/}" -q | head -1) + basename "$(docker exec "${COMPUTE}" crictl inspectp "${sandbox}" | grep -m1 -oE '/var/run/netns/cni-[0-9a-f-]+')" +} + +FRESH_NETNS=$(pod_netns "${FRESH_POD}") +FRESH_BASE=$(pod_exec "${FRESH_POD}" sh -c "ip -6 -o addr show dev eth0 scope global | awk '{print \$4}' | head -1") +FRESH_BASE=${FRESH_BASE%/*} +FRESH_LEN=96 +echo 0 >"${STATE}/fresh_n" + +# fresh prints a tenant address no pin has seen: a new address on the fresh +# tenant's interface, inside its attachment prefix, numbered from a counter +# kept in a file because fresh runs in a command substitution. Each is +# removed at exit. A failure to add the address ends the run. +fresh() { + local n addr + n=$(($(cat "${STATE}/fresh_n") + 1)) + echo "${n}" >"${STATE}/fresh_n" + addr=$(python3 -c 'import ipaddress,sys +b = bytearray(ipaddress.IPv6Address(sys.argv[1]).packed) +b[12] = 0x77; b[13] = 0x77; b[14] = int(sys.argv[2]) >> 8; b[15] = int(sys.argv[2]) & 0xff +print(ipaddress.IPv6Address(bytes(b)))' "${FRESH_BASE}" "${n}") + if ! docker exec "${COMPUTE}" ip netns exec "${FRESH_NETNS}" ip -6 addr add "${addr}/${FRESH_LEN}" dev eth0 nodad >&2; then + echo "FAIL: could not add fresh tenant address ${addr}" >&2 + kill -TERM $$ + return 1 + fi + echo "${addr}" >>"${STATE}/fresh" + echo "${addr}" +} + +# --- probes ----------------------------------------------------------------------- + +# probe KIND TENANT SRC COUNT ALLOWED... sends COUNT probes of KIND (tcp6, +# tcp4, udp6, udp4, icmp6 or icmp4) from TENANT, sourced from SRC (or the +# tenant's own address when SRC is "-"), toward the off-fabric host, and +# checks every one succeeded and arrived from an address in ALLOWED. The +# addresses seen are written to ${STATE}/seen. It fails, with the reason, on +# any failed request, on a capture that did not see every probe, and on any +# unexpected source. +probe() { + local kind="$1" tenant="$2" src="$3" count="$4" + shift 4 + local allowed=("$@") dst filter cmd cap ok=0 seen + local opt_tcp="" opt_udp="" opt_ping="" + if [ "${src}" != "-" ]; then + opt_tcp="--interface ${src}" opt_udp="-s ${src}" opt_ping="-I ${src}" + fi + case "${kind}" in + tcp6 | udp6 | icmp6) dst=${HOST6} ;; + *) dst=${HOST4_SYNTH} ;; + esac + case "${kind}" in + tcp6) filter="ip6 and dst ${HOST6} and tcp dst port 80 and ip6[53] & 0x12 == 0x02" ;; + tcp4) filter="ip and dst ${HOST4} and tcp dst port 80 and tcp[13] & 0x12 == 0x02" ;; + udp6) filter="ip6 and dst ${HOST6} and udp dst port 33434" ;; + udp4) filter="ip and dst ${HOST4} and udp dst port 33434" ;; + icmp6) filter="icmp6 and dst ${HOST6} and ip6[40] == 128" ;; + icmp4) filter="icmp and dst ${HOST4} and icmp[0] == 8" ;; + esac + case "${kind}" in + tcp*) cmd="for i in \$(seq ${count}); do curl -s -o /dev/null -w '%{http_code}\n' --max-time 4 ${opt_tcp} 'http://[${dst}]/'; done" ;; + udp*) cmd="for i in \$(seq ${count}); do echo probe | nc -u -w1 ${opt_udp} ${dst} 33434; done; echo sent" ;; + icmp*) cmd="ping -6 -c ${count} -i 0.3 -W 2 ${opt_ping} ${dst}" ;; + esac + cap="${STATE}/cap.$$" + docker exec "${REMOTE}" timeout $((count * 4 + 15)) tcpdump -i eth1 -nn -l -c "${count}" "${filter}" >"${cap}" 2>&1 & + local tcpdump_pid=$! + sleep 2 + local out + out=$(pod_exec "${tenant}" sh -c "${cmd}" 2>&1) || true + local cap_rc=0 + wait "${tcpdump_pid}" || cap_rc=$? + case "${kind}" in + tcp*) ok=$(grep -c '^200$' <<<"${out}" || true) ;; + udp*) ok=${count} ;; # judged by arrival below; UDP has no answer to count + icmp*) ok=$(sed -n 's/.* \([0-9]*\) received.*/\1/p' <<<"${out}" | head -1) ;; + esac + # A TCP or UDP source is printed as address.port; ICMP has no port, and an + # IPv4 address's last octet must not be taken for one. + local strip='s/\.[0-9]+$//' + case "${kind}" in icmp*) strip='s/^//' ;; esac + seen=$(grep -oE '^[0-9:.]+ IP6? [0-9a-f:.]+ >' "${cap}" | awk '{print $3}' | + sed -E "${strip}" | sort -u || true) + local captured + captured=$(grep -cE '^[0-9:.]+ IP6? ' "${cap}" || true) + printf '%s\n' "${seen}" | sed '/^$/d' >"${STATE}/seen" + rm -f "${cap}" + if [ "${ok:-0}" != "${count}" ]; then + echo " ${kind} from ${tenant} (${src}): ${ok:-0}/${count} succeeded" >&2 + probe_diagnostics "${tenant}" "${src}" "${out}" + return 1 + fi + if [ "${cap_rc}" -ne 0 ] || [ "${captured}" -lt "${count}" ]; then + echo " ${kind} from ${tenant} (${src}): the host saw ${captured}/${count} probes" >&2 + return 1 + fi + local addr a found + while read -r addr; do + [ -n "${addr}" ] || continue + found=0 + for a in "${allowed[@]}"; do + if [ "${addr}" = "${a}" ]; then found=1; fi + done + if [ "${found}" -eq 0 ]; then + echo " ${kind} from ${tenant} (${src}): arrived from ${addr} ($(shard_of "${addr}")), want one of ${allowed[*]}" >&2 + return 1 + fi + done <"${STATE}/seen" + return 0 +} + +# probe_diagnostics prints what a failed probe needs to be understood: the +# tenant's output, whether its source address exists, and how galactic-cni's +# groups stood. +probe_diagnostics() { + [ -n "${QUIET_DIAG:-}" ] && return 0 + { + echo " output: $(tr '\n' ' ' <<<"$3")" + if [ "$2" != "-" ]; then + docker exec "${COMPUTE}" ip netns exec "${FRESH_NETNS}" ip -6 -o addr show to "$2/128" | + awk '{print " address present: " $4}' + fi + docker exec "${COMPUTE}" curl -s --max-time 4 "${CNI_METRICS}" | + grep -E '^galactic_cni_egress_(pool_members|shard_selections_total|group_routes|shard_watch_synced)' | + sed 's/^/ /' + } >&2 +} + +# expect is probe that ends the run on failure. +expect() { + probe "$@" || die "$1 probe failed" +} + +# seen_count prints how many distinct sources the last probe saw. +seen_count() { wc -l <"${STATE}/seen" | tr -d ' '; } +seen_first() { head -1 "${STATE}/seen"; } + +# --- configuration, saved and restored ------------------------------------------ + +ENV_VARS=(GALACTIC_CNI_EGRESS_MODE GALACTIC_CNI_EGRESS_HASH GALACTIC_CNI_EGRESS_PIN_IDLE GALACTIC_CNI_EGRESS_POOL_MIN_ACTIVE) +declare -A SAVED_ENV +env_of() { + kc -n galactic-system get ds galactic-cni -o json | + python3 -c 'import json,sys +d = json.load(sys.stdin) +for c in d["spec"]["template"]["spec"]["containers"]: + if c["name"] == "credential-refresh": + for e in c.get("env", []): + if e["name"] == sys.argv[1]: print(e.get("value", "")); sys.exit(0) +print("")' "$1" +} +for v in "${ENV_VARS[@]}"; do SAVED_ENV[${v}]=$(env_of "${v}"); done +declare -A SAVED_DRAIN +for s in "${SHARDS[@]}"; do + SAVED_DRAIN[${s}]=$(kc -n galactic-system get egressshard "${s}" -o jsonpath='{.spec.drain}') +done +kc get clusterrole galactic-cni -o json >"${STATE}/clusterrole.json" + +# replace_clusterrole [drop-egressshards] replaces galactic-cni's ClusterRole +# with the saved copy, optionally without the egressshards rule. +replace_clusterrole() { + python3 -c 'import json,sys +d = json.load(open(sys.argv[1])) +for k in ("resourceVersion", "uid", "creationTimestamp", "managedFields"): d["metadata"].pop(k, None) +if len(sys.argv) > 2: + d["rules"] = [r for r in d["rules"] if "egressshards" not in r.get("resources", [])] +json.dump(d, sys.stdout)' "${STATE}/clusterrole.json" "$@" | docker exec -i "${CP}" kubectl replace -f - >/dev/null +} + +rollout_cni() { + kc -n galactic-system rollout status daemonset/galactic-cni --timeout=900s >/dev/null + # The metrics endpoint answers once the new process has started. + local deadline=$((SECONDS + 120)) + until [ "$(metric "galactic_cni_egress_mode{mode=\"$1\"}")" = 1 ]; do + [ ${SECONDS} -lt ${deadline} ] || die "galactic-cni on ${COMPUTE} does not report mode $1" + sleep 2 + done +} + +set_mode() { # MODE HASH PIN_IDLE + kc -n galactic-system set env daemonset/galactic-cni -c credential-refresh \ + GALACTIC_CNI_EGRESS_MODE="$1" GALACTIC_CNI_EGRESS_HASH="$2" GALACTIC_CNI_EGRESS_PIN_IDLE="$3" \ + GALACTIC_CNI_EGRESS_POOL_MIN_ACTIVE=2 >/dev/null + rollout_cni "$1" +} + +restore_env() { + local args=() v + for v in "${ENV_VARS[@]}"; do + if [ "${SAVED_ENV[${v}]}" = "" ]; then args+=("${v}-"); else args+=("${v}=${SAVED_ENV[${v}]}"); fi + done + kc -n galactic-system set env daemonset/galactic-cni -c credential-refresh "${args[@]}" >/dev/null +} + +set_drain() { # SHARD true|false|"" + local value=null + [ -n "$2" ] && value=$2 + kc -n galactic-system patch egressshard "$1" --type=merge -p "{\"spec\":{\"drain\":${value}}}" >/dev/null +} + +PROHIBITED=() +prohibit() { + docker exec "${COMPUTE}" ip -6 route add prohibit "$1" + PROHIBITED+=("$1") +} +unprohibit() { + docker exec "${COMPUTE}" ip -6 route del prohibit "$1" 2>/dev/null || true +} + +WITHDRAWN="" +frr() { # SHARD vtysh-args... + local node pod + node=${NODE[$1]} + shift + pod=$(kc -n galactic-system get pods -o wide | awk -v n="${node}" '/fabric-router/ && $7 == n {print $1}') + kc -n galactic-system exec "${pod}" -c frr -- vtysh "$@" +} +originate() { # SHARD yes|no + local no="" + [ "$2" = no ] && no="no " + frr "$1" -c 'conf t' -c 'router bgp 65000' -c 'address-family ipv6 unicast' -c "${no}network $(sid64 "$1")" >/dev/null +} + +cleanup() { + local rc=$? + set +e + for p in "${PROHIBITED[@]}"; do unprohibit "${p}"; done + [ -n "${WITHDRAWN}" ] && originate "${WITHDRAWN}" yes + if [ -f "${STATE}/fresh" ]; then + while read -r a; do + docker exec "${COMPUTE}" ip netns exec "${FRESH_NETNS}" ip -6 route del prohibit "${HOST6}/128" from "${a}" 2>/dev/null + docker exec "${COMPUTE}" ip netns exec "${FRESH_NETNS}" ip -6 addr del "${a}/${FRESH_LEN}" dev eth0 2>/dev/null + done <"${STATE}/fresh" + fi + for s in "${SHARDS[@]}"; do set_drain "${s}" "${SAVED_DRAIN[${s}]}"; done + replace_clusterrole + docker exec "${REMOTE}" rm -f /var/www/localhost/htdocs/active-active-soak + restore_env + kc -n galactic-system rollout status daemonset/galactic-cni --timeout=900s >/dev/null + rm -rf "${STATE}" + exit "${rc}" +} +trap cleanup EXIT +# A failure inside a command substitution (fresh) ends the run with TERM; +# exiting normally on it runs cleanup. +trap 'exit 143' TERM + +# --- background download ---------------------------------------------------------- + +# soak_start TENANT starts a slow download from TENANT and records which shard +# its connection left from. soak_check waits for it and fails unless it +# completed intact. +SOAK_PID="" +SOAK_SHARD="" +soak_start() { + local cap="${STATE}/soakcap" + docker exec "${REMOTE}" timeout 20 tcpdump -i eth1 -nn -l -c 1 \ + "ip6 and dst ${HOST6} and tcp dst port 80 and ip6[53] & 0x12 == 0x02" >"${cap}" 2>&1 & + local tcpdump_pid=$! + sleep 2 + pod_exec "$1" curl -s -o /dev/null -w '%{size_download} %{exitcode}' --limit-rate 150k --max-time 300 \ + "http://[${HOST6}]/active-active-soak" >"${STATE}/soak" 2>"${STATE}/soak.err" & + SOAK_PID=$! + wait "${tcpdump_pid}" || die "could not see the download's connection at the host" + SOAK_SHARD=$(shard_of "$(grep -oE 'IP6 [0-9a-f:.]+ >' "${cap}" | awk '{print $2}' | sed -E 's/\.[0-9]+$//')") + [ "${SOAK_SHARD}" != unknown ] || die "the download left from no shard address: $(cat "${cap}")" +} +soak_check() { + local rc=0 got="" code="" + wait "${SOAK_PID}" || rc=$? + read -r got code <"${STATE}/soak" || true + if [ "${rc}" -ne 0 ] || [ "${got}" != "${SOAK_BYTES}" ] || [ "${code}" != 0 ]; then + die "$1: the download ended at ${got:-nothing} of ${SOAK_BYTES} bytes (curl ${code:-none}, exec ${rc}): $(cat "${STATE}/soak.err")" + fi +} + +docker exec "${REMOTE}" dd if=/dev/urandom of=/var/www/localhost/htdocs/active-active-soak bs=1M count=${SOAK_MB} 2>/dev/null + +# ================================================================================== + +log "--- 0. preflight ---" +for s in "${SHARDS[@]}"; do + for cond in Ready Programmed; do + st=$(kc -n galactic-system get egressshard "${s}" -o jsonpath="{.status.conditions[?(@.type==\"${cond}\")].status}") + [ "${st}" = True ] || die "${s} ${cond}=${st:-missing}" + done + [ -n "${V6[${s}]}" ] && [ -n "${V4[${s}]}" ] || die "${s} lacks an IPv6 or IPv4 masquerade address" + frr "${s}" -c "show bgp ipv6 unicast $(sid64 "${s}")" | grep -q best || die "${s}'s SID is not originated" + docker exec "${TRANSIT}" vtysh -c "show bgp ipv6 unicast ${V6[${s}]}" | grep -q best || + die "transit has no route back to ${s}'s ${V6[${s}]}" + docker exec "${TRANSIT}" vtysh -c "show bgp ipv4 unicast ${V4[${s}]}" | grep -q best || + die "transit has no route back to ${s}'s ${V4[${s}]}" +done +if [ "${V6[${SHARDS[0]}]}" = "${V6[${SHARDS[1]}]}" ] || [ "${V4[${SHARDS[0]}]}" = "${V4[${SHARDS[1]}]}" ] || + [ "$(sid64 "${SHARDS[0]}")" = "$(sid64 "${SHARDS[1]}")" ]; then + die "the shards share a SID locator or masquerade address" +fi +schema=$(kc get crd egressshards.network.datumapis.com \ + -o jsonpath='{.spec.versions[?(@.name=="v1alpha1")].schema.openAPIV3Schema.properties.spec.properties.drain.type}') +[ "${schema}" = boolean ] || die "the EgressShard CRD has no spec.drain; install the network CRD that adds it" +set_drain "${SHARDS[1]}" true +[ "$(kc -n galactic-system get egressshard "${SHARDS[1]}" -o jsonpath='{.spec.drain}')" = true ] || + die "spec.drain does not persist: the API server pruned it" +set_drain "${SHARDS[1]}" "${SAVED_DRAIN[${SHARDS[1]}]}" +[ "$(kc auth can-i watch egressshards.network.datumapis.com -n galactic-system \ + --as=system:serviceaccount:galactic-system:galactic-cni)" = yes ] || die "galactic-cni may not watch EgressShards" +log "ok shards distinct and routed back, spec.drain persists, galactic-cni may watch EgressShards" + +# Self-test: a blackholed fresh tenant must fail the probe the drain check uses. +a=$(fresh) +docker exec "${COMPUTE}" ip netns exec "${FRESH_NETNS}" ip -6 route add prohibit "${HOST6}/128" from "${a}" +if QUIET_DIAG=1 probe tcp6 "${FRESH_POD}" "${a}" 3 "${V6[@]}" 2>/dev/null; then + die "self-test: a blackholed tenant passed the probe; the checks below would prove nothing" +fi +docker exec "${COMPUTE}" ip netns exec "${FRESH_NETNS}" ip -6 route del prohibit "${HOST6}/128" from "${a}" +expect tcp6 "${FRESH_POD}" "${a}" 3 "${V6[@]}" +log "ok self-test: a blackholed fresh tenant fails the probe, the same tenant unblocked passes" + +log "--- 1. hashed, flow hash, pins off ---" +set_mode hashed flow 0 +wait_metric "$(members nat66 active)" 2 120 +wait_metric "$(members "${NAT64_CLASS}" active)" 2 120 +for kind in tcp6 tcp4 udp6 udp4; do + case "${kind}" in *6) allowed=("${V6[@]}") ;; *) allowed=("${V4[@]}") ;; esac + expect "${kind}" "${FRESH_POD}" - 24 "${allowed[@]}" + [ "$(seen_count)" = 2 ] || die "24 ${kind} flows from one tenant used $(seen_count) shard(s), want both" + log "ok ${kind}: one tenant's flows used both shards" +done +expect icmp6 "${FRESH_POD}" - 4 "${V6[@]}" +expect icmp4 "${FRESH_POD}" - 4 "${V4[@]}" +log "ok icmp6 and icmp4 answered through a shard" +for s in "${SHARDS[@]}"; do + base=$(python3 -c 'import ipaddress,sys; b=bytearray(ipaddress.IPv6Address(sys.argv[1]).packed); b[8]&=0xF0; b[9]=0; print(ipaddress.IPv6Address(bytes(b)))' "${SID[${s}]}") + for class in nat66 "${NAT64_CLASS}"; do + p=$(metric "galactic_cni_egress_shard_packets_total{class=\"${class}\",shard_sid=\"${base}\"}") + awk -v p="${p:-0}" 'BEGIN {exit !(p > 0)}' || die "no ${class} packets counted toward ${s}" + done +done +log "ok galactic-cni counted NAT66 and NAT64 packets toward both shards" + +log "--- 2. hashed, source hash, default pins ---" +set_mode hashed source 2h4m +wait_metric "$(members nat66 active)" 2 120 +declare -A SHARD6 SHARD4 +for t in "${TENANTS[@]}"; do + expect tcp6 "${t}" - 6 "${V6[@]}" + [ "$(seen_count)" = 1 ] || die "${t}'s NAT66 connections left from $(seen_count) addresses, want one" + SHARD6[${t}]=$(shard_of "$(seen_first)") + expect tcp4 "${t}" - 6 "${V4[@]}" + [ "$(seen_count)" = 1 ] || die "${t}'s NAT64 connections left from $(seen_count) addresses, want one" + SHARD4[${t}]=$(shard_of "$(seen_first)") + log "ok ${t}: NAT66 via ${SHARD6[${t}]}, NAT64 via ${SHARD4[${t}]}" +done +expect udp6 "${FRESH_POD}" - 4 "${V6[${SHARD6[${FRESH_POD}]}]}" +expect icmp6 "${FRESH_POD}" - 3 "${V6[${SHARD6[${FRESH_POD}]}]}" +expect udp4 "${FRESH_POD}" - 4 "${V4[${SHARD4[${FRESH_POD}]}]}" +expect icmp4 "${FRESH_POD}" - 3 "${V4[${SHARD4[${FRESH_POD}]}]}" +log "ok ${FRESH_POD}'s UDP and ICMP leave from the same shard as its TCP, per family" + +A=${FRESH_POD} +SHARD_A=${SHARD6[${A}]} +B="" +for t in "${TENANTS[@]}"; do + if [ "${SHARD6[${t}]}" != "${SHARD_A}" ]; then + B=${t} + break + fi +done +[ -n "${B}" ] || die "every tenant on ${COMPUTE} hashed to ${SHARD_A}; cannot test a failure between shards" +SHARD_B=${SHARD6[${B}]} + +log "--- 3. galactic-cni restart keeps sessions and placement ---" +soak_start "${A}" +kc -n galactic-system rollout restart daemonset/galactic-cni >/dev/null +rollout_cni hashed +soak_check "galactic-cni restart" +for t in "${TENANTS[@]}"; do + expect tcp6 "${t}" - 3 "${V6[${SHARD6[${t}]}]}" +done +log "ok a download survived the restart and every tenant kept its shard" + +log "--- 4. [prohibit route] ${SHARD_B} unreachable from ${COMPUTE} ---" +soak_start "${A}" +[ "${SOAK_SHARD}" = "${SHARD_A}" ] || die "${A}'s download left from ${SOAK_SHARD}, want its pinned ${SHARD_A}" +loc_b=$(locator "${SHARD_B}") +prohibit "${loc_b}" +wait_metric "$(members nat66 active)" 1 30 +expect tcp6 "${B}" - 3 "${V6[${SHARD_A}]}" +expect tcp6 "${FRESH_POD}" "$(fresh)" 3 "${V6[${SHARD_A}]}" +soak_check "${SHARD_B} unreachable" +unprohibit "${loc_b}" +wait_metric "$(members nat66 active)" 2 30 +log "ok ${B} moved to ${SHARD_A}, a fresh tenant went there, ${A}'s download on ${SHARD_A} completed" + +if [ "${SKIP_BGP:-0}" != 1 ]; then + log "--- 5. [real BGP] ${SHARD_B}'s SID withdrawn by its node ---" + WITHDRAWN=${SHARD_B} + start=${SECONDS} + originate "${SHARD_B}" no + wait_metric "$(members nat66 unreachable)" 1 300 + log " withdrawal reached the group in $((SECONDS - start))s" + for _ in 1 2 3; do expect tcp6 "${FRESH_POD}" "$(fresh)" 2 "${V6[${SHARD_A}]}"; done + start=${SECONDS} + originate "${SHARD_B}" yes + WITHDRAWN="" + wait_metric "$(members nat66 active)" 2 300 + log "ok fresh tenants went to ${SHARD_A} while ${SHARD_B}'s SID was withdrawn; re-origination took $((SECONDS - start))s" +else + log "--- 5. [real BGP] skipped (SKIP_BGP=1) ---" +fi + +log "--- 6. drain ${SHARD_A} ---" +soak_start "${A}" +[ "${SOAK_SHARD}" = "${SHARD_A}" ] || die "${A}'s download left from ${SOAK_SHARD}, want ${SHARD_A}" +set_drain "${SHARD_A}" true +wait_metric "$(members nat66 draining)" 1 60 +expect tcp6 "${A}" - 3 "${V6[${SHARD_A}]}" +log " ${A}, still active, keeps its pin on draining ${SHARD_A}: a busy tenant keeps a shard draining" +for _ in 1 2 3 4; do + expect tcp6 "${FRESH_POD}" "$(fresh)" 2 "${V6[${SHARD_B}]}" +done +expect tcp4 "${FRESH_POD}" "$(fresh)" 2 "${V4[${SHARD_B}]}" +soak_check "drain" +set_drain "${SHARD_A}" "${SAVED_DRAIN[${SHARD_A}]}" +wait_metric "$(members nat66 draining)" 0 60 +log "ok every fresh tenant went to ${SHARD_B}; ${A}'s download completed on draining ${SHARD_A}" + +log "--- 7. [prohibit routes] every shard unavailable ---" +empty_before=$(metric 'galactic_cni_egress_shard_selections_total{result="group_empty"}') +for s in "${SHARDS[@]}"; do prohibit "$(locator "${s}")"; done +wait_metric "$(members nat66 active)" 0 30 +if QUIET_DIAG=1 probe tcp6 "${FRESH_POD}" "$(fresh)" 2 "${V6[@]}" 2>/dev/null; then + die "a fresh tenant reached the internet with every shard unavailable" +fi +empty_after=$(metric 'galactic_cni_egress_shard_selections_total{result="group_empty"}') +awk -v a="${empty_after:-0}" -v b="${empty_before:-0}" 'BEGIN {exit !(a > b)}' || + die "the drops were not counted as group_empty (${empty_before:-0} -> ${empty_after:-0})" +for s in "${SHARDS[@]}"; do unprohibit "$(locator "${s}")"; done +wait_metric "$(members nat66 active)" 2 30 +expect tcp6 "${FRESH_POD}" "$(fresh)" 2 "${V6[@]}" +log "ok no egress and group_empty counted while no shard was reachable; egress back once they were" + +log "--- 8. EgressShard watch failure ---" +replace_clusterrole drop-egressshards +kc -n galactic-system rollout restart daemonset/galactic-cni >/dev/null +rollout_cni hashed +sleep 15 +[ "$(metric galactic_cni_egress_shard_watch_synced)" = 0 ] || die "the watch reports synced without permission" +[ "$(metric 'galactic_cni_egress_pool_enabled{class="nat66"}')" = 1 ] || die "the published group was lost" +for t in "${A}" "${B}"; do expect tcp6 "${t}" - 2 "${V6[@]}"; done +replace_clusterrole +wait_metric galactic_cni_egress_shard_watch_synced 1 300 +log "ok without its watch galactic-cni said so and kept forwarding; the watch synced once permitted again" + +log "--- 9. rollback to ordered mode ---" +restore_env +rollout_cni ordered +deadline=$((SECONDS + 180)) +until [ "$(metric_sum galactic_cni_egress_group_routes)" = 0 ] && + [ "$(metric_sum galactic_cni_egress_pool_enabled)" = 0 ]; do + [ ${SECONDS} -lt ${deadline} ] || die "routes still name a group, or a group is still enabled, after 180s" + sleep 3 +done +first=${SHARDS[0]} +for t in "${TENANTS[@]}"; do + expect tcp6 "${t}" - 2 "${V6[${first}]}" + expect tcp4 "${t}" - 2 "${V4[${first}]}" +done +log "ok no route names a group, the groups are retired, every tenant is back on ${first}" + +echo "PASS: hashed egress on dfw: both shards in both families, restart, local and BGP failure, drain, total outage, watch outage and rollback" diff --git a/docs/agents/ARCHITECTURE-CNI.md b/docs/agents/ARCHITECTURE-CNI.md index dea3d0e6..7965a508 100644 --- a/docs/agents/ARCHITECTURE-CNI.md +++ b/docs/agents/ARCHITECTURE-CNI.md @@ -265,8 +265,9 @@ Constraints below for the manifest). plus the static conflist. - `run` — `--grpc-health-port` flag (default `5180`), calls `installer.Run(ctx, grpcHealthPort)`: serves gRPC health checks, manages the eBPF uSID datapath's - load/attach lifecycle, and periodically refreshes the kubeconfig token and - rotates the CNI log file. + load/attach lifecycle, keeps tenant VRFs' egress routes on their egress + shards (ordered or hashed mode, `GALACTIC_CNI_EGRESS_MODE`), and + periodically refreshes the kubeconfig token and rotates the CNI log file. `--build-info` and `--version`/`-V` utility flags are also present on the root command, same as every other binary. @@ -518,6 +519,7 @@ auto-detect directly (see Configuration above). - **Every VRF table ends in an unreachable default, per family.** `vrf.Add` installs `unreachable default` for IPv4 and IPv6 at metric `4278198272` (`vrf.UnreachableDefaultMetric`, the value the kernel's VRF documentation recommends), on every call, not only when it creates the VRF. Without it, a lookup that misses in the VRF's table falls through the rules after the l3mdev rule to the main table, and the host routes a tenant packet the eBPF datapath didn't claim out its own default interface with the tenant's source address. The high metric means any real default in the table, such as the ingress sidecar's, still wins. Route sweepers must leave it alone: `hostgw` only removes unicast routes on the attachment's own device, and the installer's sidecar-return prune only touches tables from `0xF000` up. Each has a test pinning that. - **TCP MSS is clamped in the datapath, not by the tenant or the fabric MTU.** A tenant packet crosses the fabric inside a 40-byte outer IPv6 header with no SRH, so a tenant interface at the uplink's own MTU advertises an MSS the path cannot carry, and nothing on the path fragments or reports an oversized segment. `clamp_tcp_mss` in `usid.c` lowers the MSS of every SYN and SYN-ACK at three points: `usid_egress` just before it encapsulates, `usid_egress`'s DSR reply redirect (the client's segments reach the backend encapsulated by the gateway), and `usid_ingress` just after it decapsulates (the only place a SYN-ACK returning through an egress shard can be clamped, since the shard does not). The limits are per tenant family, 1400 for IPv6 and 1420 for IPv4 at a 1500-byte fabric, and the IPv6 value also covers NAT64. `galactic-cni run` sizes them from the smallest uplink MTU (`internal/plumbing/ebpf/mssclamp`) into `mss_clamp_table`, overridable with `GALACTIC_CNI_TCP_MSS_CLAMP` (see [docs/cni/environment-variables.md](../cni/environment-variables.md#galactic_cni_tcp_mss_clamp)). The map and its counters (`mss_clamp_stats`, sized with spare slots) are new maps rather than new fields or `drop_reasons` slots, because changing an existing pinned map's layout makes `attach.Load` recreate that map empty, and the rows only CNI ADD writes do not come back until each attachment is re-added. `clamp_tcp_mss` is a global BPF function, verified once on its own, and walks at most 16 options: past about 20 steps the verifier's cost jumps from thousands of instructions to over its 1M limit. `usid_verifier_cost_test.go` fails if either program's verifier cost passes 16,000 instructions. +- **Tenant egress can spread across every egress shard in the cluster (hashed mode).** With `GALACTIC_CNI_EGRESS_MODE=hashed`, a tenant VRF's `::/0` and NAT64 entries in `egress_route_table` are not a single shard SID but a group sentinel: `link_ifindex == 0xFFFFFFFF`, the group ID in the first four bytes of `sid` and the tenant's Argument in bytes 8 and 9. There is one group per translation class, NAT66 (group 0) and each configured NAT64 prefix (groups 1-3), and a group holds only the `EgressShard`s whose status can translate its class (an IPv6 masquerade address for NAT66; an IPv4 address and that `nat64Prefix`, or `translatesWellKnownPrefix` for `64:ff9b::/96`, for NAT64), so traffic can never be placed on a shard that would drop it. `usid_egress` resolves the sentinel in `resolve_egress_group`: it looks the group up in `egress_shard_groups`, an `ARRAY_OF_MAPS` whose slot holds the group's current snapshot (one immutable `struct egress_shard_group_value`: Maglev table plus every member's SID, next hop, flags and slot generation), hashes the tenant's source address (or the 5-tuple, `GALACTIC_CNI_EGRESS_HASH=flow`) with FNV-1a over `struct egress_shard_pin_key`, follows a live pin in `egress_shard_pin` or else the Maglev table, completes the member's base SID with the Argument and encapsulates as for an ordinary entry. Every read comes from one snapshot. `egressroutemap.KernelGroupStore` publishes a change by creating a new inner map, filling it and swapping it into the outer slot in one update, so a packet sees the old snapshot or the new one and the kernel frees the old one after RCU; an ordinary array value updated in place was measured to hand out torn records (17 of 55,000 packets mixed one shard's SID with another's MACs under concurrent updates), the snapshot swap none. `egress_shard_group_template`, an unused instance of the inner map type, exists only so clang older than 21 emits the value's full BTF. A pin is honoured only while its slot still carries the generation it recorded; generations come from `next_slot_generation`, published in the snapshot, and an in-process high-water mark that also covers failed publications, so a slot's generation is never reused (see `ShardGroupTable`'s doc comment and `docs/nat/configuration.md#how-the-groups-are-published`). The groups are written only by `galactic-cni run` (`internal/installer/egressgroup.go`): it watches the cluster's `EgressShard`s (a candidate needs `status.shardSID` with `Programmed` and `Ready` true and no deletion timestamp; `spec.drain` makes it draining), resolves each SID to an uplink neighbor (only a resolvable shard is alive), and rebuilds on any EgressShard, route or neighbor change as well as the 30 s sweep. CNI ADD (`attachreg.installVRFEgress`, also used by the datapath repair's `FillVRF`) writes each route to its class's group when any group is open, skips a class with no eligible shard, and fails like an unresolvable ordered list when a class's shards are all unreachable. Leaving hashed mode is a three-step protocol: the sweep marks every group `CLOSING` (still forwarding, but CNI ADD writes no new sentinel), then `EgressRouteTable.Refresh` moves every sentinel back to the first resolvable `GALACTIC_CNI_EGRESS_SHARD_SIDS` entry, and the groups are disabled only after a sweep that read and wrote every route without error and found none left; an ADD that read the groups open re-checks after writing and rewrites its routes in ordered mode if they closed, so no sentinel is written after the last scan. `galactic_cni_egress_group_routes` reports the sentinels left. An empty or disabled group drops (`egress_shard_stats`, `galactic_cni_egress_shard_selections_total`), never fails open. Privileged tests run the datapath under BPF_PROG_RUN against snapshots the Go side published: placement against `ShardHash` and `internal/maglev` (`shardgroup_datapath_test.go`), publication faults at each step, restart, class isolation and concurrent swaps against live packets (`shardgroup_publication_test.go`). See [docs/nat/configuration.md](../nat/configuration.md#active-active-egress). - **A packet too big for the fabric gets an ICMP error from the datapath, not a silent drop at the uplink.** `usid_egress` calls `send_too_big` after the MSS clamp and before it encapsulates. A packet longer than `encap_mtu_table`'s limit (the uplink MTU less 40, so 1460 at a 1500-byte fabric) is rewritten in place into an ICMPv6 Packet Too Big, or an ICMPv4 Fragmentation Needed for an IPv4 tenant with DF set, and redirected back out the interface it arrived on. The error carries the limit, comes from the attachment's own IPAM gateway (`tenant_gw_table`, keyed by host-side ifindex and written by `galactic-bgp` at ADD, removed by `galactic-veth`/`galactic-tap` at DEL), and quotes the packet's first 68 bytes with NPTv6 undone, so the tenant's path MTU discovery matches it to the socket that sent it. A GSO packet is judged by its segment length. No error is sent for an IPv4 packet without DF, for an ICMP error, past a per-CPU token bucket of 1,000 a second with a burst of 100 (`pmtu_icmp_bucket`), or for an attachment with no gateway in that family; those are dropped and counted as `fib_frag_needed`. Each outcome is counted in `pmtu_stats`, exported as `galactic_usid_pmtu_packets_total{result}`, and the limit as `galactic_usid_pmtu_limit_bytes`. The same `mssclamp.Reconciler` writes `encap_mtu_table` and `mss_clamp_table` from one `GALACTIC_CNI_TCP_MSS_CLAMP` setting, so the two cannot disagree. A pod's or guest's TCP and UDP packets usually arrive `CHECKSUM_PARTIAL`, which no BPF helper can clear, and a device without checksum offload (a tap whose guest did not negotiate it) would finish that checksum into the middle of the error. So the error is laid out with room pushed at the front until the kernel's checksum spot (the original transport header plus 16 for TCP, 6 otherwise) lands on a trailing slot word, with a pad word chosen so the sum the kernel writes there equals what is already there. `usid_pmtu_tap_test.go` checks this against the kernel through a `virtio_net_hdr` tap, with offload on and off. - **An uplink that drops out of auto-detection keeps `usid_ingress` for 60 seconds.** `attach.Watch` re-resolves the uplink set on every netlink link or route event, and an uplink found only through BGP-learned routes leaves that set whenever FRR withdraws them, which it does for every restart (#778). Detaching on the first miss dropped SRv6 traffic still arriving on that bond for 6 to 14 seconds. An attached interface whose link still exists now keeps its hook, and its place in the published uplink list, until it has been missing for `detachHoldDown`. Watch re-evaluates on its own when the hold-down ends, so an uplink that really stops carrying fabric routes is still detached. A deleted link is dropped at once. - **CRD-driven config, no sidecar gRPC.** `galactic-bgp` writes `BGPVRFInstance`/`BGPAdvertisement` CRDs directly via a bare controller-runtime client; `galactic-router`'s reconciler picks them up. No in-node gRPC calls between any of the CNI-chain binaries and `galactic-router`. @@ -577,6 +579,7 @@ auto-detect directly (see Configuration above). | How `galactic-bgp`/`galactic-route` learn state without touching the kernel | `internal/cnibgp/prevresult.go:inferFromPrevResult` (reads `RawPrevResult`, not the dead `PrevResult` field) | | Host gateway address/route configuration (shared by both master plugins) | `internal/hostgw/hostgw.go:ConfigureHostGateway` | | CNI DaemonSet install/refresh | `internal/installer/installer.go:Bootstrap` (init container), `internal/installer/installer.go:Run` (long-running container) | +| Egress shard placement (ordered and hashed mode) | `internal/installer/egressgroup.go`, `internal/plumbing/ebpf/egressroutemap/shardgroup.go`, `usid.c:resolve_egress_group` | | Interface naming / base62 encoding | `internal/plumbing/intf/intf.go` | **Stable vs. frequently changed:** diff --git a/docs/agents/ARCHITECTURE-GATEWAY.md b/docs/agents/ARCHITECTURE-GATEWAY.md index ded873e4..4f4fe1ee 100644 --- a/docs/agents/ARCHITECTURE-GATEWAY.md +++ b/docs/agents/ARCHITECTURE-GATEWAY.md @@ -916,7 +916,7 @@ that tag into `config/galactic-gateway/base`. **Stable vs. frequently changed:** - Stable: `internal/maglev/table.go` (a settled, well-tested algorithm — Google's published Maglev construction), `internal/plumbing/ebpf/edgemap` (mirrors `usidmap`'s already-settled crash-safety pattern) - Active: `internal/gateway/quota.go`/`telemetry.go` (real but coarse — see Key Design Decisions; likely to grow richer enforcement) -- Out of scope here, but related and evolving: `galactic-nat`'s sharded stateful egress translation tier (`cmd/galactic-nat`, a separate binary; unlike this gateway, it has no consistent-hash ring for shard selection today — `EgressDefaultRouteAdd` installs only the first resolvable shard SID, every other configured shard sitting as cold standby) +- Out of scope here, but related and evolving: `galactic-nat`'s sharded stateful egress translation tier (`cmd/galactic-nat`, a separate binary). Its compute side has its own Maglev ring, built with the same `internal/maglev` package but never shared with this gateway's: in hashed egress mode `usid_egress` places each tenant address on one of the cluster's shards (see [ARCHITECTURE-CNI.md](ARCHITECTURE-CNI.md#key-design-decisions)); in the default ordered mode every VRF uses the first resolvable shard SID and the rest stand by **Non-obvious patterns:** - `gatewayDatapathKeepAlive` (`cmd/galactic-gateway/gateway.go`) intentionally never calls `Close` on the loaded eBPF objects or the XDP `link.Link` — see that var's doc comment for the live incident this guards against (silent GC-triggered detach with every control-plane signal still looking healthy). diff --git a/docs/cni/environment-variables.md b/docs/cni/environment-variables.md index f2e1eb25..4b4a17fc 100644 --- a/docs/cni/environment-variables.md +++ b/docs/cni/environment-variables.md @@ -197,7 +197,7 @@ of 100; refusals are counted as `rate_limited`. Comma-separated list of the `galactic-nat` shards' `Status.ShardSID`s this node's tenant VRFs egress through — `internal/plumbing/srv6.EgressDefaultRouteAdd` installs a VRF's default -egress route toward the first one that resolves, at CNI ADD. It is +egress route toward the first one that resolves, at CNI ADD, in ordered mode. It is operator-supplied since no single Kubernetes CRD is visible across this multi-cluster fabric's separate clusters/API servers the way BGP itself is. Shards run on edge nodes; set this per site to that site's own edge shards, @@ -227,9 +227,63 @@ Because the destination therefore differs per tenant, a shard advertises its SID's covering `/64` rather than a host route; see `EgressShardReconciler.shardAdvertisementPrefixes`. +In hashed egress mode (`GALACTIC_CNI_EGRESS_MODE=hashed`, below) this list +does not decide placement. The cluster's `EgressShard`s do, and the list is +read only to move VRFs back onto it when switching to ordered mode. + **Type:** comma-separated string · **Default:** _(empty — no shard configured)_ +## `GALACTIC_CNI_EGRESS_MODE` + +How tenant VRFs reach the cluster's egress shards. Read by `galactic-cni run` +(the `credential-refresh` container) only; CNI ADD follows whatever group it +has written into the pinned datapath maps. + +- `ordered` sends each VRF to the first shard in + `GALACTIC_CNI_EGRESS_SHARD_SIDS` that resolves. The rest stand by. +- `hashed` spreads tenants across every healthy `EgressShard` in the + cluster: `usid_egress` picks a shard per tenant address from a Maglev table + every compute node builds identically. See + [Active/active egress](../nat/configuration.md#active-active-egress). + +A value other than these stops the container at startup, and so does +`hashed` with more than three entries in `GALACTIC_CNI_NAT64_PREFIX`: there +is one shard group for NAT66 and one for each NAT64 prefix, four in all. + +**Type:** `ordered` or `hashed` · **Default:** `ordered` + +## `GALACTIC_CNI_EGRESS_HASH` + +What hashed mode hashes. `source` keeps every connection from one tenant +address on one shard, and so on one public address. `flow` hashes the +5-tuple, which spreads one heavy address over every shard but shows it to the +internet from several public addresses. Read by `galactic-cni run` only. + +**Type:** `source` or `flow` · **Default:** `source` + +## `GALACTIC_CNI_EGRESS_PIN_IDLE` + +How long hashed mode keeps a tenant address on the shard its first packet +went to, after its last packet, as a Go duration. Adding or draining a shard +then leaves established sessions where they are. A tenant that keeps sending +keeps its pin, and so keeps a draining shard in use, for as long as it +sends. `0` turns pins off, and with it any way for a drain to preserve +sessions. Pins are per compute node, at most 65536. Read by `galactic-cni run` +only. + +**Type:** duration (`0`, or at least `1s`) · **Default:** `2h4m`, the egress +shard's established-TCP idle timeout + +## `GALACTIC_CNI_EGRESS_POOL_MIN_ACTIVE` + +The fewest active egress shards the cluster should have. Exported as +`galactic_cni_egress_pool_min_active` for `GalacticEgressPoolBelowMinimum` +to compare against; it does not change forwarding. Read by `galactic-cni run` +only. + +**Type:** non-negative integer · **Default:** `1` + ## `GALACTIC_CNI_DAN_DIR` Directory the tap master plugin writes a sandbox's Directly Attachable diff --git a/docs/feature_reviews/egress-active-active_2026-10-09-resolution.md b/docs/feature_reviews/egress-active-active_2026-10-09-resolution.md new file mode 100644 index 00000000..c3328cec --- /dev/null +++ b/docs/feature_reviews/egress-active-active_2026-10-09-resolution.md @@ -0,0 +1,108 @@ +# Resolution: egress-active-active review (2026-10-09) + +Resolution record for [egress-active-active_2026-10-09.md](egress-active-active_2026-10-09.md), +which reviewed `f61103b2`, `69ebb6cc` and `c8830ea9`. The review is kept as +written; this file records what was done about each finding and how it was +validated. + +## Findings + +| Finding | Status | Resolution | +| ---------------------------------- | ------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| F01 Disable after a failed refresh | Fixed | `egressGroup.refresh` returns `Refresh`'s error. Leaving hashed mode publishes every enabled group as `CLOSING` (forwarding, but CNI ADD writes no new sentinel), then converts, and disables only after a conversion that read and wrote every route without error and found no sentinel left, foreign-namespace sentinels included. CNI ADD re-reads the groups after writing its routes and rewrites them in ordered mode if they closed meanwhile, which closes the race with concurrent registration. | +| F04 Pool compatibility | Fixed | One shard group per translation class: NAT66 for `::/0`, one per NAT64 prefix (at most three in hashed mode; more is a startup error). A shard joins a class's group only if its status can translate it (IPv6 address; or IPv4 address and that `nat64Prefix`, or `translatesWellKnownPrefix` for `64:ff9b::/96`). Every sweep recomputes eligibility, so a shard that changes leaves. Ineligible shards are logged and exported (`pool_members{state="ineligible"}`); a class no shard serves gets no route and `GalacticEgressClassUnserved`. Single-family clusters keep working. | +| F05 Generation uniqueness | Fixed | Superseded by the F06 design: a group is published as one snapshot in one atomic swap, so a slot that failed to publish was never visible to a packet. Generations come from `next_slot_generation` in the published snapshot plus an in-process high-water mark that also counts failed publications; disable keeps the counter, restart resumes from it, 0 is never handed out. The review's A→B(failed)→C sequence is reproduced against the kernel at each of the three publication steps. | +| F06 Coherent BPF publication | Fixed | `egress_shard_group_table` and `egress_shard_table` (ARRAY maps updated in place) replaced by `egress_shard_groups`, an `ARRAY_OF_MAPS` whose slot holds one immutable snapshot: Maglev table and every member's SID, next hop, flags and generation. Publication creates a new inner map, fills it and swaps the outer slot in one update (an RCU pointer replacement; the old snapshot is freed after readers finish). The packet path reads everything from that one snapshot; one extra map lookup per grouped packet. The finding was confirmed on a kernel before the fix: updating a published value in place produced 17 torn packets out of 54,777; the snapshot swap produced 0 out of 49,011. | +| F02 Drain verification | Fixed | `verify-nat-active-active.sh` rewritten. Every probe requires every request to succeed, the capture to see every probe, and every source to be an expected masquerade address; silence fails. Fresh, never-pinned tenant addresses are added inside the tenant's network namespace, so drain and failure are checked under default pins. A self-test blackholes a fresh tenant and requires the probe to fail. Background downloads propagate failures; cleanup restores the environment, drain flags, ClusterRole, routes and addresses it found. | +| F03 Drain lifecycle documentation | Fixed | Docs and the new rollout checklist say: `spec.drain: true`, wait for sessions to end, then delete; deletion tears the shard down at once; a busy tenant can keep a shard draining indefinitely; pins off cannot preserve sessions. A terminating `EgressShard` is now excluded from the groups rather than treated as draining. No finalizer was added. | + +## Additional recommendations + +| Recommendation | Resolution | +| --------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| CRD/schema prerequisite, RBAC, drain read-back | Documented in `docs/nat/configuration.md` (Prerequisites) and `docs/nat/active-active-rollout.md`; the lab check verifies all three. | +| Unique identities, translation compatibility, return routing | Documented as prerequisites; enforced for compatibility (F04); the lab preflight checks distinct SIDs and addresses and a return route to each from the transit. | +| NAT64 and TCP/UDP/ICMP coverage | The lab check covers TCP, UDP and ICMP over NAT66 and NAT64. | +| Default pins, addition, withdrawal, all down, restart, informer, rollback | Covered by the lab check (addition by a shard's re-origination), except addition of a third shard, which the lab topology cannot do; it is covered by the privileged datapath tests. | +| Prohibit route vs BGP | Lab stages are labelled; the real BGP stage removes the SID origination from the shard node's FRR. | +| Load, pins, replication, fragments | Documented in configuration.md (How a shard is chosen; Known constraints). | +| Cross-node placement | Confirmed: VRF table IDs are allocated per node (`vrf.nextFreeTableID`). Kept, and documented as a per-node guarantee: sessions, pins and reply SIDs are node-specific, so a stable cross-node identity would buy nothing. | +| Monitoring of zero candidates, informer failure, incompatibility, missing metrics | New control-loop series (`egress_mode`, `shard_watch_synced`, `group_routes`, `sweep_errors_total`) and alerts (`GalacticEgressClassUnserved`, `GalacticEgressShardWatchNotSynced`, `GalacticEgressSweepFailing`, `GalacticCNIMetricsDown`); `PoolEmpty` fires on zero candidates with routes; an unpublished node reports no pool series rather than zeros. | +| Central evaluator | Documented: repository rules are the tested source; infra must install them in its central evaluator. | +| Staging acceptance and rollback checklist | `docs/nat/active-active-rollout.md`. | + +## Compatibility + +- **Map layout.** The two pre-review group maps are gone; `egress_shard_groups` + and `egress_shard_group_template` are new. No map that existed before the + feature changes, so an upgrade from a release without it keeps every pinned + row. A node that ran a pre-review build of this branch (only the + containerlab lab) has the two obsolete pins removed at load; its groups are + rebuilt by the next sweep, and until then a route naming a group drops. +- **Rollback.** Unchanged in principle and now observable: switch to ordered + mode, wait for `galactic_cni_egress_group_routes` to read 0 on every node, + then roll back the image. +- **Older clang.** `egress_shard_group_template` exists because clang before + 21 emits the inner map value as a forward declaration. Generation was + checked with clang-18 (CI's compiler) and clang-19 (the image builder's). + +## Validation + +Run on 2026-10-09 on a Fedora 44 host (kernel 7.1) against the containerlab +`gvpc` lab's dfw cluster. The lab is local and isolated; no staging or +production resource was touched. + +| Check | Result | +| ------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `golangci-lint` (via `task ci`) | Passed, 0 issues. | +| `yamlfmt -lint` | Passed with a temporary config. `task ci`'s own yamlfmt step stops in a git worktree on `stat .git/**: not a directory`, a known worktree artifact; the remaining `ci` stages were then run directly. | +| `task build`, `task test:unit`, `task test:e2e` | Passed. The unprivileged unit run skips every root-gated test. | +| `task test:unit-root` (privileged, host clang 22) | Passed in two of three full runs. The third failed once in `internal/cnibgp` (`TestRegisterEBPFDatapath_RegistersAllThreeTables`: its test interface was not found), which then passed 3 of 3 alone and 5 of 5 as a package on this branch and on `origin/main`; that code is unchanged here. A later run reproduced it, with `internal/cni`'s `TestCmdCheckValidConfigMissingResources`, only while root packages ran in parallel; with `-p 1` every root-gated package passed, so these tests collide on host interfaces when run concurrently. Skipped: `TestReconcileRadvActors_FailedRemovalIsRetried`, as before. | +| Privileged datapath tests against clang-18 objects | Passed: the whole `prog` suite (verifier acceptance and the verifier-cost limit included) and every `TestGroupDatapath`/`TestGroupPublication` test, with objects built by clang-18, the CI compiler. Generation also checked with clang-19, the image builder's compiler. | +| Concurrent publication (`ConcurrentSwapsStayCoherent`) | 0 mixed packets of 122,312 across 400 swaps. A throwaway control that rewrote a published value in place produced 17 mixed of 54,777 (not committed: it is nondeterministic by nature). | +| `task test:alerts` | Passed: 32 rules, every rule test. | +| `verify:nat-active-active` (lab) | Passed all ten stages, including the real BGP withdrawal (withdrawal and re-origination each reached the group in about 2 s). An earlier run failed once in the drain stage when a fresh tenant got no answer; it was not reproduced by hand or in the next full run after the script's fresh-address allocation was made unique (it had reused a random byte across a subshell). The cause is not proven. | +| Upgrade on lab nodes | `galactic-cni` loaded the new maps beside every existing pin and removed the two obsolete ones on all three dfw nodes. | + +Not run: the privileged suite on the staging kernel version, any staging +acceptance step, adding a third shard in a live cluster (the lab has two per +site), and infra's central alert evaluation. + +## Remaining staging gates + +1. Release the `network` change adding `spec.drain` (branch + `feat/egressshard-drain`, commit `36ee528`), install its CRD in staging, + and re-pin galactic's `go.mod` to the release. +2. infra: install the CNI RBAC, originate every shard's SID, IPv6 and IPv4 + addresses from its own node with a distinct IPv4 per shard, copy the + `galactic-cni-egress` rule group to the central evaluator, and write the + runbook anchors for the seven egress alerts. +3. Run `docs/nat/active-active-rollout.md` in staging, including the + privileged CI job on the staging kernel version. + +## Risks + +- One extra map lookup per grouped packet, and a new inner map per change to + a group (about 2.4 KiB, freed after RCU); a flapping shard costs a + publication per change. +- A busy tenant keeps a draining shard in use indefinitely; draining needs an + operator decision on how long to wait. +- The cause of the one unreproduced drain-stage lab failure is unknown. + +## Re-review findings + +[egress-active-active_2026-10-09_rereview.md](egress-active-active_2026-10-09_rereview.md) +reviewed an intermediate state of the fixes above. Status of its findings +against the code as committed: + +| Finding | Status | Notes | +| ---------------------------------------------------------- | ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| R01 Closing does not exclude in-flight CNI writes | Partial | CNI ADD re-reads the groups after writing and rewrites its routes in ordered mode if they closed. Every ordered-mode sweep also moves any route still naming a group back to the static list, so a late write is repaired within one sweep (30s), dropping that VRF's egress meanwhile. There is no cross-process barrier, so `group_routes` reading 0 can precede such a write by one sweep. | +| R02 Empty hashed membership restores static shard routes | Open | With no class served, a hashed sweep refreshes without groups and moves routes back to the first resolvable static SID, bypassing readiness and translation checks. | +| R03 Unserved NAT64 prefix falls through to the NAT66 route | Open | CNI ADD skips a configured NAT64 prefix with no eligible shard, so longest-prefix match sends it to the NAT66 group's `::/0` route. | +| R04 Fresh-address counter lost in subshells | Fixed | The counter is kept in a state file, and a failed address add ends the run. | +| R05 Group IDs reused when the prefix list changes | Open | Group IDs are positions in `GALACTIC_CNI_NAT64_PREFIX`; removing or reordering a prefix republishes a group that existing routes still name for a different class. | +| R06 BGP stage withdraws only one route source | Not borne out | The stage removes only the FRR origination, and the shard was reported unreachable within about 2s on every passing run: the compute resolver did not resolve the SID through the EVPN plain route. The stage does not separately assert the compute FIB lookup. | + +R02, R03 and R05 affect hashed mode only and block enabling it; ordered mode, +the default, does not read the groups. diff --git a/docs/feature_reviews/egress-active-active_2026-10-09.md b/docs/feature_reviews/egress-active-active_2026-10-09.md new file mode 100644 index 00000000..9b1b90c1 --- /dev/null +++ b/docs/feature_reviews/egress-active-active_2026-10-09.md @@ -0,0 +1,246 @@ +# Architecture Review: egress-active-active +**Language:** go | **Reviewed:** 2026-10-09T20:36:23.650844+00:00 + +## Summary + +Not ready for staging enablement signoff: fix rollback and shard-state publication defects, enforce pool compatibility, and complete privileged validation. + +**6 findings:** 🔴 0 critical 🟠 3 high 🟡 3 medium ⚪ 0 low + + +**Scope:** `f61103b2`, `69ebb6cc`, `c8830ea9` (`HEAD~3..HEAD`). Working tree was clean when review began. No earlier report exists in this directory. Review covers implementation, added tests, configuration, monitoring, docs and lab verification. No staging resources were changed. + +**Decision:** Hold staging enablement signoff. The feature can proceed to an isolated validation environment after the high-priority fixes; this review does not certify the live staging environment or current CI run. Changes are opt-in and the deployment manifest still defaults to ordered mode. Estimated remediation is **medium**, principally because coherent BPF publication needs design and kernel validation. + +### Validation evidence + +| Check | Result and limit | +| --- | --- | +| `task build:ebpf` | Passed; all four datapaths regenerated, including both BPF endiannesses. Compilation does not prove kernel verifier acceptance. | +| `go build ./cmd/...` | Passed (exit 0); emitted a nonfatal warning about a read-only module stat cache. | +| `bin/golangci-lint run --timeout 5m` | Passed: `0 issues.` | +| Focused `go test -race -count=1 -v` | `egressroutemap`, `attachreg`, `metrics` passed. Root-gated cases skipped, including **all seven new TestGroupDatapath tests**. | +| Config and added installer tests | Passed. The full installer suite encountered sandbox restrictions on TCP/netlink sockets; this is not evidence of an application regression or a passing full suite. | +| Focused `go vet` | Passed for config, installer, egressroutemap, attachreg and metrics. | +| Failure-injection reproduction | Confirmed F05 in a temporary fake-map test: A starts at generation 1; replacing with B writes slot generation 2 but group publication fails; replacing with C also gets generation 2. Test removed after execution; source implementation unchanged. | +| Shell syntax; diff whitespace | `bash -n` and `git diff --check HEAD~3 HEAD` passed. | +| YAML formatter | Could not complete: `stat .git/**: not a directory` in this worktree. | +| Privileged BPF, containerlab, alert tests, full CI | Not completed. Docker socket access denied and review runs unprivileged. No promtool executable available. | +| Remote CI status | Unverified: GitHub API connection failed. | +| Live staging CRD, routing, scrape health and image availability | Unverified; no live staging evidence obtained. | + +Logs from this review are in `/tmp/galactic-review-{datapath-tests,reproductions,lint,build}.log` for this workspace session. + +### Staging acceptance gates + +1. Fix F01 and F05 with iterator/write failure-injection tests. Resolve F06 with an explicit publication/snapshot contract and concurrency coverage. For F04, enforce compatible pool membership; a tightly controlled initial staging trial can also explicitly verify homogeneous shards, but this does not fix general admission behavior. +2. Install and verify the external `EgressShard` CRD version supporting `spec.drain` (the Go dependency now points to network commit `36ee528904ff`). A Go module bump does not update a cluster's schema. Verify a drain patch persists by reading it back. Confirm the updated CNI get/list/watch RBAC is deployed. Document these prerequisites in the rollout procedure. +3. Verify every candidate has unique advertised SID and masquerade addresses, supports the required NAT66/NAT64 families, and agrees on NAT64 prefix/well-known-prefix behavior. Test return routing through **each** shard before distributing tenants. +4. Require a green `task ci`, the separate privileged `task test:unit-root` job on the target kernel, and `task test:alerts`. Fix F02 before treating `verify:nat-active-active` as an acceptance gate. +5. Run a two-shard staging canary using `hashed`, `source`, the default `2h4m` pin idle, and pool minimum `2`. Exercise NAT66 **and NAT64**, TCP/UDP/ICMP, large packets and sustained sessions. The added lab script uses only IPv6 HTTP/NAT66; its flow phase uses pins off and its source phase uses 20 seconds, so it does not establish all default-mode behavior. +6. Check shard addition, actual BGP route withdrawal and restoration, explicit drain, CNI restart with existing pins, initial informer delay/API outage, all shards unavailable, and concurrent membership changes. A prohibit route validates local selection failure, not full BGP failure detection timing. Confirm sessions on a surviving shard continue; sessions on a failed shard are expected to break because NAT state is not replicated. +7. Verify the existing CNI PodMonitor produces `job="galactic-cni"` and correct node labels, the new rules are installed in the actual evaluator (infra centrally evaluates its own copy), and targets scrape successfully. Confirm active/minimum, per-shard traffic and drop counters. Do not interpret absent series as health. +8. Rehearse rollback: preserve the static SID list in the init-container configuration, switch to ordered on the new binary, verify no sentinel routes remain and the group is disabled, then roll back the binary. Mode changes can break sessions placed on a different shard. Delete only **after explicit drain and session quiescence**, not as the drain operation. + +## Architecture + +**Layers observed:** environment configuration → installer informer and netlink watch → egress shard group reconciliation → BPF map adapters → Prometheus collectors + +**Dependency direction:** clean + +``` +EgressShard informer + netlink events -> debounced installer sweep -> shard maps + per-VRF route maps -> datapath +``` + +Reviewed the combined HEAD~3..HEAD change (33 files, 4,237 insertions and 161 deletions), surrounding CNI/NAT controller, resolver, loader, CI and deployment contracts. Dependency direction is coherent. See validation and staging gates below for limits of the evidence. + +## Findings + +### 🔴 P0 — Critical + +No P0 findings. + +### 🟠 P1 — High + +#### [F01] Do not disable the group after a failed ordered-mode refresh +**Location:** `internal/installer/egressgroup.go:175-179` | **Category:** error-handling + +**Problem:** g.refresh logs and discards Refresh's error, returning partial counters. Refresh decides all sentinel rewrites first and increments Ungrouped, leaving InGroup zero when every destination resolves. If iteration or any subsequent Put fails, some/all sentinel routes remain but ordered sweep still disables their group. This violates the promised rollback ordering. + +**Risk:** Switching hashed to ordered during a transient map write/read failure drops all packets on the unconverted VRFs until a later successful sweep; sustained error makes the outage persistent. + +**Before:** +``` +result := g.refresh(table, foreignTableIDs, g.staticSIDs) +if result.InGroup == 0 { groups.Disable(egressroutemap.ClusterShardGroup) } +``` +**After:** +``` +result, err := g.refresh(table, foreignTableIDs, g.staticSIDs) +if err == nil && result.InGroup == 0 { groups.Disable(egressroutemap.ClusterShardGroup) } +``` + +#### [F04] Validate shard translation compatibility before adding it to the pool +**Location:** `internal/installer/egressgroup.go:310-326` | **Category:** config + +**Problem:** Candidate selection accepts every Ready+Programmed SID without checking status.shardAddressIPv6, status.shardAddressIPv4, or status.nat64Prefix. The API/controller legitimately allows shards serving only one family; identityFromSpec only requires at least one masquerade address (egressshard_controller.go:394-397). Both default NAT66 and NAT64 routes now use this same pool. Adding a Ready NAT66-only shard therefore directs a fraction of NAT64 requests to a shard unable to translate them; a different NAT64 prefix has the same defect. Ordered mode could explicitly select compatible shards, but hashed mode silently broadens to every shard. + +**Risk:** Partial deterministic tenant egress outage when a cluster contains valid but heterogeneous shards, including incremental rollout of an incompletely configured second shard. + +**Before:** +``` +if shard.Status.ShardSID == "" || !Programmed || !Ready { continue } +candidates = append(candidates, ShardCandidate{SID: sid, Draining: ...}) +``` +**After:** +``` +Validate each candidate against the compute node's required NAT66/NAT64 families and NAT64 prefix; reject incompatible pool configuration with a surfaced error, or maintain eligible groups per translation route. +``` + +#### [F05] Reserve slot generations across failed group publication +**Location:** `internal/plumbing/ebpf/egressroutemap/shardgroup.go:291,356-361` | **Category:** concurrency + +**Problem:** A new slot receives current group generation + 1, but slots are written before the group generation is committed. If the group Put fails, the changed slot is already visible to packets through the old Maglev table. Another candidate replacement before the next successful group publication receives the same generation. Pins created for the first replacement are then accepted for the second. The documented never-reused identity guarantee is false on partial failure. + +**Risk:** A pin can silently follow an unrelated shard after partial reconciliation; draining replacements can inherit live pins and existing sessions can move unexpectedly. + +**Before:** +``` +nextGeneration := current.Generation + 1 +value.Generation = nextGeneration +// write slots, then group; return on failure +``` +**After:** +``` +Allocate identities from durable monotonic state that includes partially published slots, and preserve uniqueness through retries and slot clearing. Add failure-injection tests across every write boundary. +``` + +### 🟡 P2 — Medium + +#### [F02] Require successful new-tenant egress in the drain check +**Location:** `deploy/containerlab/scripts/verify-nat-active-active.sh:224-231` | **Category:** testing + +**Problem:** The drain test only rejects observed packets on the drained shard. sources suppresses curl/tcpdump errors, and an empty capture produces seen=[]; the loop executes zero times and announces success. A regression blackholing all unpinned tenants during drain therefore passes this stage, while the existing A connection alone is checked for success. + +**Risk:** The advertised end-to-end staging gate can certify broken drain behavior. + +**Before:** +``` +mapfile -t seen < <(sources "${t}" 3) +for addr in "${seen[@]}"; do ...; done +echo "ok ..." +``` +**After:** +``` +Require successful requests and exactly one captured source, matching the surviving shard's expected masquerade address, before reporting success. +``` + +#### [F03] Remove the claim that deleting a shard gracefully drains it +**Location:** `docs/nat/configuration.md:517-518` | **Category:** config + +**Problem:** The new operator guide says deleting EgressShard drains it like spec.drain. EgressShardReconciler has no finalizer, withdraws advertisement on deletion (internal/controller/egressshard_controller.go:167-185), and excludes terminating shards from syncNode at line 220. Translation is cleared immediately rather than retained until sessions drain. Merely marking a terminating object as a draining candidate cannot change this lifecycle. + +**Risk:** An operator using deletion as documented interrupts every session on that shard. + +**Before:** +``` +Deleting an EgressShard drains it the same way for as long as it takes to go. +``` +**After:** +``` +Set spec.drain=true, wait for sessions to reach zero, then delete. Deleting without this sequence immediately tears down the shard and its sessions. +``` + +#### [F06] Read a coherent shard snapshot during concurrent map updates +**Location:** `internal/plumbing/ebpf/prog/usid.c:1212-1228,3673-3680,3712-3736` | **Category:** concurrency + +**Problem:** The group and shard maps are ordinary BPF ARRAY maps. Userspace updates modify the existing value in place, while the packet program retains pointers and reads flags/generation separately from SID, ifindex and MACs. A slot or next-hop update can interleave after validation but before route copying, so a packet can validate one identity and encapsulate with a different or mixed record. Ordered writes between maps do not make an individual array record atomic. + +**Risk:** Transient packet loss or incorrect encapsulation during membership and next-hop changes. Existing sequential BPF_PROG_RUN tests cannot establish concurrency safety. This is source-supported, not reproduced on a kernel in this review. + +**Before:** +``` +shard = bpf_map_lookup_elem(...); // validate flags/generation +... +memcpy(route.sid, shard->sid, ...); route.link_ifindex = shard->link_ifindex; +``` +**After:** +``` +Publish immutable/versioned records or use a supported synchronization scheme that copies and validates one consistent snapshot; stress updates concurrently with real packet processing. +``` + +### ⚪ P3 — Low + +No P3 findings. + +## What's Working Well + +- Configuration validates mode, hash, pin timeout, and minimum at startup rather than silently falling back. +- Unsynced informer state is distinguished from an empty candidate set. +- Informer and netlink events feed a bounded debounced sweep; periodic ticker backs up event delivery. +- Egress route sweeps preserve sidecar-owned namespace state. +- Docs explicitly require switching to ordered before rolling back to a binary predating group sentinels. +- New read-only EgressShard RBAC is scoped by resource and informer namespace. +- Metrics expose active/draining/unreachable state and selection outcomes; existing CNI PodMonitor is present. +- Added installer helper tests and config tests pass; verification script passes bash syntax check. +- go vet ./internal/config ./internal/installer passes. + +## Refactor Plan + +Each step is independently mergeable and must not break existing behavior. + +### Step 1: Preserve refresh errors through rollback +*Addresses: F01* + +**What:** Return the error from g.refresh and only disable groups after a complete successful conversion; add failure-injection coverage for iterator and route Put errors. + +**Why now:** Rollback reliability is a staging safety prerequisite. + +**Safe when:** Failed conversion leaves group enabled and old sentinel routes usable; successful retry converts routes then disables it. + +### Step 2: Make shard publication and identity retry-safe +*Addresses: F05, F06* + +**What:** Preserve unique slot identities through partial writes and implement coherent datapath snapshots. + +**Why now:** Pin continuity depends on stable identity and consistent route reads. + +**Safe when:** Fault injection cannot reuse identity, and concurrent packet/update stress shows no mixed route or invalid pin acceptance. + +### Step 3: Enforce compatible translation identities +*Addresses: F04* + +**What:** Validate status families and NAT64 prefix against local egress requirements before pool admission, with explicit error reporting and mixed-pool tests. + +**Why now:** Ready alone does not mean the shard can serve every route moved to the common pool. + +**Safe when:** A NAT66-only or differently prefixed shard cannot receive NAT64 traffic it does not support. + +### Step 4: Make drain verification fail on absence of traffic +*Addresses: F02* + +**What:** Check request success and require the expected surviving source address. + +**Why now:** Readiness evidence must detect drops as well as wrong placement. + +**Safe when:** A blackholed fresh tenant makes the script fail; a healthy surviving shard passes. + +### Step 5: Correct deletion procedure +*Addresses: F03* + +**What:** Document explicit drain, session wait, then delete; remove unsupported graceful-delete claim. + +**Why now:** Operators need accurate lifecycle guidance before first staging rollout. + +**Safe when:** Runbook matches reconciler lifecycle and no longer treats delete as graceful. + +## Additional constraints and coverage observations + +- Source placement is address-based, not load-aware. Roughly equal address counts do not imply equal byte rates when tenants vary in traffic volume. +- Pins are local to a compute node and bounded by a 65,536-entry LRU. Eviction after a membership change can move an active address; drain can persist indefinitely if an address keeps sending. This is not replicated NAT session state. +- Hash input includes the node-local Linux VRF table ID. `internal/plumbing/vrf/vrf.go` allocates these locally, so identical Maglev member sets alone do not guarantee the same placement for a VPC address on different compute nodes. Workload migration continuity needs separate validation; no cross-node pin transfer is implemented. +- Flow-mode fragment placement is tested for IPv4 in the selector, but existing NAT translation explicitly lacks full fragment handling. Do not advertise end-to-end fragment support based on the selector test. +- Current new installer tests cover candidate filtering, event matching and kick coalescing, not informer startup, actual sweep sequencing, partial failures or mode-transition races. Datapath tests are sequential; they do not test userspace/kernel update interleavings. +- The group-empty alert intentionally requires at least one member. A zero-candidate pool with existing sentinels instead relies on below-minimum or traffic-driven drop alerts. An informer that never initializes can leave enabled=0, suppressing the pool alerts; verify desired mode and startup logs separately. + +For F06, the kernel documents that array lookup returns a pointer into shared array storage and requires synchronization for concurrent access: [Linux BPF array-map documentation](https://docs.kernel.org/bpf/map_array.html). `array_map_update_elem` copies into the existing element unless locking is explicitly requested: [Linux array-map implementation](https://github.com/torvalds/linux/blob/master/kernel/bpf/arraymap.c). These establish the map semantics; the packet-level interleaving is an inference from this change's reads and writes, not an observed live incident. diff --git a/docs/feature_reviews/egress-active-active_2026-10-09_rereview.md b/docs/feature_reviews/egress-active-active_2026-10-09_rereview.md new file mode 100644 index 00000000..12d1484f --- /dev/null +++ b/docs/feature_reviews/egress-active-active_2026-10-09_rereview.md @@ -0,0 +1,230 @@ +# Architecture Review: egress-active-active +**Language:** go | **Reviewed:** 2026-10-09T21:36:38.278824+00:00 + +## Summary + +Not ready for staging enablement: snapshot publication is substantially improved, but rollback and translation-class routing still have correctness defects. + +**6 findings:** 🔴 0 critical 🟠 4 high 🟡 2 medium ⚪ 0 low + + +**Scope:** Current uncommitted changes relative to `c8830ea9`, including new untracked implementation files. HEAD has not advanced since the original review. Reviewed against the original report and the implementation prompt. The original report is preserved unchanged. + +**Decision:** Hold staging enablement. The snapshot redesign addresses the original mutable-array and partial-publication problems, but four high-priority routing/transition findings remain. Estimated remediation is medium. No implementation files were changed by this review and no infrastructure was modified. + +### Original findings disposition + +| Original finding | Assessment | +| --- | --- | +| F01 — unsafe rollback after refresh errors | Error propagation fixed. Concurrent CNI registration is still unsafe (R01). | +| F02 — drain test accepts no traffic | Original vacuous pass fixed. Fresh-address allocation and the real-BGP failure scenario need correction (R04, R06). | +| F03 — deletion described as graceful drain | Fixed in the updated guide and deletion candidate filtering. | +| F04 — incompatible pool members | Per-class capability filtering added, but route selection can bypass it (R02, R03), and class reconfiguration can redirect to a different class (R05). | +| F05 — generation reuse after partial writes | Structurally addressed by atomic snapshot publication and persisted slot generation state. | +| F06 — torn shard records during updates | Structurally addressed by immutable map-in-map publication. Concurrent kernel test added; execution remains unverified here. | + +### Validation and reproductions + +- `task build:ebpf`: passed, all four datapaths regenerated. +- `go build ./cmd/...`: passed (exit 0), with a nonfatal read-only module stat-cache warning. +- Repository-wide `golangci-lint run --timeout 5m`: passed, **0 issues**. +- `go test -race -count=1 -v` for egressroutemap, attachreg and metrics: passed for runnable cases. Privileged packet/publication tests skipped, including the new concurrent-swap and publication-failure tests. +- Focused installer group/shard/egress/class tests: passed. +- Focused `go vet` for installer, egressroutemap, attachreg, metrics and attach: passed. +- `bash -n` for the acceptance script and `git diff --check`: passed. +- Three temporary Go overlay reproductions confirmed R01, R02 and R05 without modifying source files. These tests assert the defective behavior to establish the finding; their passing is not acceptance evidence. +- R01: inject a sentinel after the ordered sweep's scan. The group becomes disabled with one referencing route left. +- R02: a synced empty shard list converts three existing group routes back to a resolvable static shard. +- R05: remove the first NAT64 prefix from a two-prefix configuration. Its route remains on group 1, which now serves the second prefix and its different shard. +- R03 follows directly from the installed route set and longest-prefix-match lookup; existing single-family tests expect the missing specific prefix and do not exercise actual packet selection through the catch-all. +- No privileged kernel acceptance, containerlab scenario, promtool alert tests, full `task ci`, remote CI verification or live staging checks were completed in this review. A green lint/unit run does not close these gates. + +Reproduction artifacts for this session: `/tmp/review-overlay.json`, `/tmp/review_regression_test.go`, `/tmp/review-class-overlay.json`, `/tmp/review_class_lifecycle_test.go`, and `/tmp/egress-rereview-class-repro.log`. Other checks are logged under `/tmp/egress-rereview-*.log`. + +## Architecture + +**Layers observed:** installer informer → translation-class groups → CNI registration → metrics → acceptance script + +**Dependency direction:** clean + +Translation-scoped groups are a sound direction; empty-class and transition semantics require correction. + +## Findings + +### 🔴 P0 — Critical + +No P0 findings. + +### 🟠 P1 — High + +#### [R01] Closing groups does not exclude in-flight CNI sentinel writes +**Location:** `internal/plumbing/ebpf/attachreg/egressgroups.go:91-106` | **Category:** concurrency + +**Problem:** A CNI ADD can read an open group and pause; ordered sweep closes it, scans/converts all visible routes and disables it; ADD then writes a sentinel referencing the disabled group. The post-write check only repairs afterward and may fail, encounter an empty static list, or never run if the process terminates. An error after a partial class install also returns before rechecking. Existing VRF routes can be affected by ADD/FillVRF, not just a not-yet-connected pod. + +**Risk:** Rollback can still blackhole tenant routes while reporting no group routes and a disabled group. Original refresh error propagation is fixed but the requested concurrent-registration safety is not. + +**Before:** +``` +installGroupRoutesIn(...); stillOpen, err := groups.AnyOpen(); installOrderedEgress(...) +``` +**After:** +``` +Serialize sentinel publication and group closing/last scan with a cross-process registration barrier or equivalent protocol; retiring must wait for admitted writers. Test late publication and failed repair. +``` + +#### [R02] Empty hashed membership silently restores static shard routes +**Location:** `internal/installer/egressgroup.go:335-339` | **Category:** config + +**Problem:** With no class having candidates, byClass is empty and WithShardGroups is omitted. Refresh interprets this as ordered mode and rewrites existing sentinels to the first resolvable static SID, even though the informer rejected it or removed its capabilities. Static SID resolution does not validate readiness or translation. + +**Risk:** Hashed mode bypasses its admission and empty-group safeguards, sending traffic to unready, deleted-but-still-routable or incompatible shards. Group-empty metrics/alerts can also lose their route evidence. + +**Before:** +``` +if len(byClass) > 0 { opts = append(opts, WithShardGroups(groupFor)) } +``` +**After:** +``` +Always express hashed refresh semantics, including an empty class map; preserve or install explicit empty-class sentinels rather than invoking ordered fallback. +``` + +#### [R03] Skipping an unserved NAT64 prefix lets NAT66 default route catch it +**Location:** `internal/plumbing/ebpf/attachreg/egressgroups.go:155-158` | **Category:** config + +**Problem:** When NAT66 has candidates but a configured NAT64 /96 has none, installGroupRoutesIn installs ::/0 and skips the /96. LPM lookup then selects the NAT66 group for packets to that configured NAT64 prefix. TestInstallVRFEgress_SingleFamilyDeployments/NAT66Only and MismatchedPrefix explicitly encode this unsafe route set. + +**Risk:** NAT64 traffic is sent to shards not admitted to its translation class, defeating F04. New VRFs behave differently from existing VRFs whose empty NAT64 sentinel remains. + +**Before:** +``` +if !found || !status.Open() || status.Candidates == 0 { continue } +``` +**After:** +``` +Install an explicit rejecting/empty-group route for each configured but unserved NAT64 prefix, or fail attachment when the class is required. Exercise actual LPM selection against ::/0. +``` + +#### [R05] Do not reuse a translation-class group while old routes reference it +**Location:** `internal/installer/egressgroup.go:355-358; internal/plumbing/ebpf/egressroutemap/egressroute.go:767-771` | **Category:** config + +**Problem:** Group IDs are the positions in the current configured prefix list. With [NAT66, P1, P2] published, removing P1 changes the list to [NAT66, P2] and overwrites group 1 with P2 before repairing existing routes. A route for P1 still names group 1. Refresh finds no configured class for P1 and deliberately leaves its sentinel untouched, so it now sends P1 traffic to P2-only shards. Old group 2 is never retired and remains a duplicate P2 group; FindClass and the sweep can pick different duplicates. + +**Risk:** Configuration changes cause persistent cross-class routing and NAT64 outages. Removed classes also remain published and can retain stale members indefinitely. + +**Before:** +``` +for i, class := range g.classes { groups.Apply(uint32(i), ..., eligible) } +``` +**After:** +``` +Allocate stable class identities and reconcile retired classes explicitly. Do not reuse a group ID until every reference has been safely migrated or replaced with a fail-closed route. Test removal, reorder, replacement, and restart with old snapshots. +``` + +### 🟡 P2 — Medium + +#### [R04] Fresh-address counter is lost in command substitutions +**Location:** `deploy/containerlab/scripts/verify-nat-active-active.sh:157-165` | **Category:** testing + +**Problem:** Every fresh call is through $(fresh), which runs in a subshell. FRESH_N increments never persist in the parent, so every address has counter=1 and only one random byte distinguishes calls. About 17 calls in the full script give roughly 42% collision probability. Reused addresses still carry default-duration pins, and ip addr add failures can be hidden because the function continues to echo an address. + +**Risk:** The acceptance gate intermittently fails for address collisions or tests already-pinned addresses as fresh tenants during drain/failure; it cannot reliably certify fresh selection. + +**Before:** +``` +a=$(fresh) # fresh increments FRESH_N locally and uses random.randrange(256) +``` +**After:** +``` +Allocate monotonically unique addresses in the parent shell or persistent state file; propagate addr-add errors and verify no address reuse. Avoid command substitution for the state mutation. +``` + +#### [R06] Withdraw every route source before measuring shard-unreachable convergence +**Location:** `deploy/containerlab/scripts/verify-nat-active-active.sh:337-340,515-516` | **Category:** testing + +**Problem:** The real-BGP scenario removes only the fabric-router network statement for the shard /64. EgressShardReconciler independently advertises that same /64 as plain reachability through galactic-router EVPN (egressshard_controller.go:466-507,529 onward), and processEVPNPath installs it through the plain-route reconciler (runtime/gobgp/monitor.go:184-188). The node loopback remains reachable. The compute resolver can therefore keep finding the EVPN route and a healthy uplink neighbor; withdrawing the FRR origin alone does not establish shard unreachability. + +**Risk:** The acceptance scenario can time out waiting for unreachable=1 on a correctly configured lab; it cannot be treated as proof of end-to-end failure detection. + +**Before:** +``` +originate "${SHARD_B}" no; wait_metric ... unreachable 1 300 +``` +**After:** +``` +Use an isolated, reversible failure that removes all usable paths to that shard, or suppress both advertisement sources with reliable restoration. Assert the actual compute FIB lookup no longer resolves over a fabric uplink before measuring selection convergence. +``` + +### ⚪ P3 — Low + +No P3 findings. + +## What's Working Well + +- Refresh errors now propagate and prevent disable after failed iterator/map write. +- Candidates check realized status families, NAT64 prefix and well-known-prefix support; terminating shards excluded. +- Drain documentation correctly requires explicit drain then quiescence before delete. +- Probe checks requests and capture counts, includes blackhole negative check and NAT64/UDP/ICMP coverage. +- Adds watch synchronization, desired mode, sweep failure metrics and staging rollback checklist. +- Immutable group snapshots combine membership, Maglev, generation and forwarding fields and publish via one ARRAY_OF_MAPS replacement. +- F05 partial slot publication is eliminated because failed create/write/swap attempts do not publish the new inner map. +- New kernel tests cover failure boundaries, restart pin retention, class isolation and concurrent snapshot swaps; these still require privileged execution. + +## Refactor Plan + +Each step is independently mergeable and must not break existing behavior. + +### Step 1: Make group retirement exclude concurrent writers +*Addresses: R01* + +**What:** Implement registration barrier and fault tests. + +**Why now:** Rollback must be safe. + +**Safe when:** No late writer can outlive disabled group. + +### Step 2: Preserve fail-closed translation classes +*Addresses: R02, R03* + +**What:** Keep hashed refresh semantics for empty classes and install explicit per-prefix failures. + +**Why now:** Admission must apply to actual traffic. + +**Safe when:** Zero candidates never sends traffic to static or another class. + +### Step 3: Reconcile translation-class lifetime +*Addresses: R05* + +**What:** Keep class-to-group identity stable and safely retire removed classes and routes. + +**Why now:** Changing configuration must not reinterpret existing sentinel routes. + +**Safe when:** Removal/reorder/replacement never forwards a route to a group for a different prefix. + +### Step 4: Make fresh probes deterministic +*Addresses: R04* + +**What:** Persist unique address allocation and check errors. + +**Why now:** Acceptance must be repeatable. + +**Safe when:** Repeated calls never reuse source addresses. + +### Step 5: Make the BGP acceptance failure remove all usable paths +*Addresses: R06* + +**What:** Account for both FRR and galactic-router advertisements; verify the compute FIB and reliably restore the failure. + +**Why now:** A routing failover test must induce the failure it claims to measure. + +**Safe when:** The test proves actual loss and recovery of reachability without a local prohibit-route shortcut. + +## Remaining staging gates and review observations + +Fix R01–R05 and correct the routing failure scenario before treating the lab script as an acceptance gate. Re-run privileged tests on the staging kernel, alert tests, full CI and the complete lab scenario. Confirm the external CRD, read-back of `spec.drain`, RBAC, unique per-shard addresses, per-class eligibility and actual central alert deployment. The new rollout checklist is useful documentation, not evidence those checks were performed. + +The new script checks UDP outbound arrival but does not demonstrate translated UDP return traffic. Shard addition, initial first-sync delay and concurrent membership changes still need explicit execution evidence. Its rollback stage assumes the saved starting mode was ordered even though cleanup claims general restoration; make that precondition explicit or set ordered for the test and restore the original state separately. + +Monitoring now exposes desired mode and watch sync, which is an improvement. Missing map-derived pool series and stale successful-sweep timestamps still need an operational check: early pinned-map open failures bypass SweepError, and there is no dedicated alert consuming the successful-sweep timestamp. Confirm those cases cannot appear healthy merely because the process answers scrapes. + +Upgrade from the reviewed experimental mutable-array build is not explicitly migrated: the new outer map starts empty while existing sentinel routes/pins can remain, and obsolete map pins are removed. If that experimental build was ever enabled anywhere, require an ordered-mode transition before this upgrade or implement/test state migration. This is an additional deployment prerequisite, not a claim that the old experimental build was deployed. diff --git a/docs/nat/active-active-rollout.md b/docs/nat/active-active-rollout.md new file mode 100644 index 00000000..08f812e4 --- /dev/null +++ b/docs/nat/active-active-rollout.md @@ -0,0 +1,120 @@ +# Active/active egress: staging acceptance and rollback + +Checklist for enabling hashed egress (`GALACTIC_CNI_EGRESS_MODE=hashed`) in a +cluster, proving it there, and backing it out. How hashed mode works is in +[configuration.md](configuration.md#active-active-egress). Run every step in +staging first; production follows the same list after a release. + +Nothing here is done by a `galactic` change alone. The CRD, the RBAC, the +`galactic-cni` environment, the routing for every shard and the alert rules +are deployed by infra. + +## 1. Before enabling + +- [ ] The release fixes the open findings R02, R03 and R05 recorded in + [the review resolution](../feature_reviews/egress-active-active_2026-10-09-resolution.md#re-review-findings). + Until then hashed mode can send a class's traffic to shards that cannot + translate it. +- [ ] The release running in the cluster contains hashed mode, and the + privileged CI job (`task test:unit-root`) passed for it on the target + kernel version. A green unprivileged run skips every datapath test. +- [ ] The `EgressShard` CRD from the `network` release that adds + `spec.drain` is installed. Prove it persists: patch `spec.drain` to + its current value on one shard and read it back with + `-o jsonpath='{.spec.drain}'`. An empty answer means the API server is + pruning the field. +- [ ] `galactic-cni`'s ClusterRole grants `get`, `list` and `watch` on + `egressshards`: + `kubectl auth can-i watch egressshards.network.datumapis.com -n galactic-system --as=system:serviceaccount:galactic-system:galactic-cni` + answers `yes`. +- [ ] Every shard is `Ready` and `Programmed`, and has its own SID locator, + its own `shardAddressIPv6` and, for NAT64, its own `shardAddressIPv4`. + No two shards share any of them. +- [ ] Every shard translates what the compute nodes need: an IPv6 address for + NAT66, and for each entry of `GALACTIC_CNI_NAT64_PREFIX`, an IPv4 + address and that prefix (or `translatesWellKnownPrefix` for + `64:ff9b::/96`). A shard missing one is kept out of that class's group, + which is safe but reduces capacity; decide whether that is intended. +- [ ] Each shard's masquerade addresses are routed back to that shard from + the internet border: query the border's or transit's BGP for each + `/128` and `/32` and confirm the path leads to the shard's own node. +- [ ] `GALACTIC_CNI_EGRESS_SHARD_SIDS` stays set on `galactic-cni`'s init + container. Hashed mode does not read it, but rollback does. +- [ ] Alerting: the `galactic-cni` PodMonitor scrapes every node as + `job="galactic-cni"` with a `node` label, and the `galactic-cni-egress` + rule group from `config/monitoring/prometheusrule.yaml` is installed + in the evaluator that actually notifies (infra's central copy, not + only this repository). Each rule has its runbook anchor in infra. + +## 2. Enable + +- [ ] Set on `galactic-cni`'s `credential-refresh` container: + `GALACTIC_CNI_EGRESS_MODE=hashed`, `GALACTIC_CNI_EGRESS_HASH=source`, + leave `GALACTIC_CNI_EGRESS_PIN_IDLE` at its default `2h4m`, and set + `GALACTIC_CNI_EGRESS_POOL_MIN_ACTIVE` to the cluster's shard count. +- [ ] Roll `galactic-cni`. Expect sessions on roughly `(N-1)/N` of tenant + addresses to break once, as their VRFs move onto the groups. +- [ ] On every compute node: `galactic_cni_egress_mode{mode="hashed"}` is 1, + `galactic_cni_egress_shard_watch_synced` is 1, + `galactic_cni_egress_pool_members{state="active"}` equals the shard + count for every class, `ineligible` is what you expect, and + `galactic_cni_egress_sweep_errors_total` does not move. Missing pool + series on a node are a failure, not an idle pool. + +## 3. Prove it + +Run each with a tenant you can watch from outside, over NAT66 and NAT64, and +over TCP, UDP and ICMP where the cluster supports them. The lab's +`task verify:nat-active-active` automates most of this against the +containerlab topology and is the reference for what to check. + +- [ ] Every shard carries traffic: `galactic_cni_egress_shard_packets_total` + rises for every shard and class, and `galactic_nat_sessions` grows on + every shard node. Equal tenant counts need not mean equal bytes. +- [ ] One tenant address keeps one public address per family across TCP, + UDP and ICMP. +- [ ] Large packets and long transfers: a sustained download and a transfer + with full-size segments complete through each shard. +- [ ] Restart `galactic-cni` during a long download: it completes, and + tenants keep their shards. +- [ ] Withdraw one shard's route for real (stop its node originating the SID, + or take its BGP session down) during a download through another shard: + that download completes, the withdrawn shard's tenants move, and + `galactic_cni_egress_pool_members{state="unreachable"}` rises. Note the + time it took. A local prohibit route proves only the node's own + reaction, not BGP convergence. Sessions on the withdrawn shard break; + nothing replicates them. Restore it and confirm it rejoins. +- [ ] Drain one shard (`spec.drain: true`): new tenant addresses go to the + others, a download already on it completes, and its session count + falls. Undrain it. +- [ ] Add a shard, if the cluster has a spare: established sessions stay + where they are, and new tenant addresses start reaching it. +- [ ] Make every shard unavailable at once, in a maintenance window only: + egress stops, `GalacticEgressPoolEmpty` fires, `group_empty` drops are + counted, nothing leaves untranslated. Restore. +- [ ] Revoke the watch permission and restart `galactic-cni` on one node: + `GalacticEgressShardWatchNotSynced` fires and egress keeps working + through the groups it already had. Restore the permission. + +## 4. Remove a shard + +- [ ] Set `spec.drain: true` and confirm with a read-back. +- [ ] Wait for `galactic_nat_sessions` on its node to reach zero, or a level + you accept losing. A tenant that never goes quiet keeps it above zero. +- [ ] Only then delete the `EgressShard`. Deletion is not a drain: it tears + down the shard and every session on it at once. + +## 5. Roll back + +- [ ] `GALACTIC_CNI_EGRESS_SHARD_SIDS` is set on the init container and + lists the cluster's shards in preference order. +- [ ] Set `GALACTIC_CNI_EGRESS_MODE=ordered` (keeping the new release) and + roll `galactic-cni`. Expect sessions whose tenants move to the first + shard to break. +- [ ] On every compute node, wait for `galactic_cni_egress_group_routes` to + read 0 for every class and `galactic_cni_egress_pool_enabled` to read 0. + While a route still names a group the group keeps forwarding, and + `GalacticEgressSweepFailing` says why a sweep cannot finish. +- [ ] Only now, if needed, roll back the `galactic-cni` image to a release + without hashed mode. Rolling back earlier blackholes every VRF whose + route still names a group. diff --git a/docs/nat/configuration.md b/docs/nat/configuration.md index 5f9a548c..2a1e9de2 100644 --- a/docs/nat/configuration.md +++ b/docs/nat/configuration.md @@ -421,7 +421,10 @@ resolves at CNI ADD wins, so the list is ordered: the first entry is the active shard and any later one is taken only by an attachment made while the earlier ones cannot be resolved. With no other site's shard in the list, a site whose shards are all unreachable fails the attachment instead of -sending its egress out through another site's edge. Example (containerlab +sending its egress out through another site's edge. In hashed mode +([Active/active egress](#active-active-egress)) the cluster's `EgressShard`s +decide placement instead, and this list only matters when switching back. +Example (containerlab lab, dfw's two edge shards; sjc and iad list their single edge shard): ```yaml @@ -441,6 +444,219 @@ egress is scheduled — `internal/plumbing/srv6.EgressDefaultRouteAdd` (called from a pod's own CNI ADD) fails outright if none of the configured shard SIDs are yet resolvable. +## Active/active egress + +By default a compute node sends each tenant VRF to one shard: the first entry +of `GALACTIC_CNI_EGRESS_SHARD_SIDS` that resolves, with every later entry +standing by. Hashed mode spreads tenants across every healthy shard in the +cluster instead, so each shard carries a share of the metro's egress and +losing one costs only the sessions it held. Staging and production rollout, +and rollback, follow +[active-active-rollout.md](active-active-rollout.md). + +| Option | Environment Variable | Set on | Default | +| --------------------- | ------------------------------------- | ----------------------------------------------- | --------- | +| Egress mode | `GALACTIC_CNI_EGRESS_MODE` | `galactic-cni`'s `credential-refresh` container | `ordered` | +| Hash | `GALACTIC_CNI_EGRESS_HASH` | `galactic-cni`'s `credential-refresh` container | `source` | +| Pin idle timeout | `GALACTIC_CNI_EGRESS_PIN_IDLE` | `galactic-cni`'s `credential-refresh` container | `2h4m` | +| Pool minimum (alerts) | `GALACTIC_CNI_EGRESS_POOL_MIN_ACTIVE` | `galactic-cni`'s `credential-refresh` container | `1` | + +Unlike the shard list, these are read by `galactic-cni run`, the long-running +container, not by the init container. A value that does not parse stops the +container at startup rather than falling back to ordered mode, and so does +hashed mode with more than three NAT64 prefixes in +`GALACTIC_CNI_NAT64_PREFIX`. + +### Prerequisites + +- **The `EgressShard` CRD with `spec.drain`.** `galactic-cni` is built + against the `network` API version that adds it, but a Go dependency does + not change a cluster's schema: install the CRD from that `network` release + in every cluster first. Without it the API server prunes `spec.drain` and + a drain silently does nothing. Check by writing the field and reading it + back: + `kubectl patch egressshard -n galactic-system --type=merge -p '{"spec":{"drain":false}}'` + then `kubectl get egressshard -n galactic-system -o jsonpath='{.spec.drain}'` + must print `false`. +- **`galactic-cni` may watch `EgressShard`s.** `config/galactic-cni/rbac.yaml` + grants `get`, `list` and `watch` on `egressshards`; + `kubectl auth can-i watch egressshards.network.datumapis.com -n galactic-system --as=system:serviceaccount:galactic-system:galactic-cni` + must answer `yes`. +- **Every shard has its own identity and return route.** Each shard needs a + distinct `shardSID` locator, a distinct `shardAddressIPv6` and, for NAT64, + a distinct `shardAddressIPv4`, each originated from that shard's own node. + In ordered mode only the first shard carried traffic, so a second shard's + addresses may never have been tested: check that a reply to each of them + reaches that shard before tenants are spread onto it. + +### How a shard is chosen + +- **One group per translation class.** A compute node needs `::/0` for + NAT66 and one route per entry of `GALACTIC_CNI_NAT64_PREFIX` for NAT64. + Each route names the group for its class, and a group holds only the shards + that can translate that class: NAT66 needs a `shardAddressIPv6`; NAT64 + toward a prefix needs a `shardAddressIPv4` and that prefix as the shard's + `nat64Prefix`, or, for the Well-Known Prefix `64:ff9b::/96`, + `translatesWellKnownPrefix`. A NAT66-only shard therefore never receives + NAT64 traffic, and a shard translating another NAT64 prefix never receives + this one's. A shard that is ready but cannot serve a class is counted as + `ineligible` for it and logged. A class no shard can serve gets no route, + as an empty shard list gives none, and `GalacticEgressClassUnserved` + fires. A cluster whose shards all serve one family is a legitimate + single-family deployment. +- **The pool is the cluster.** One metro is one cluster, so a class's pool is + every eligible `EgressShard` in `galactic-cni`'s namespace. A shard is a + candidate once its status carries a `shardSID` and both `Programmed` and + `Ready` are `True`, and it is not being deleted. + `GALACTIC_CNI_EGRESS_SHARD_SIDS` is not used for placement in hashed mode. +- **A shard counts only while it is reachable.** A dead node's `EgressShard` + status goes stale, but its BGP session drops and the route to its SID is + withdrawn. A candidate takes tenants only while its SID resolves to a + neighbor on an SRv6 uplink. `galactic-cni` rebuilds the groups on every + route or neighbor change that can affect a candidate, and every 30 seconds + as a backstop, so failover takes as long as BGP takes to withdraw the + route, plus a fraction of a second. +- **The datapath picks per packet.** `usid_egress` hashes the packet and looks + the hash up in the group's 1021-slot Maglev table. When a shard joins or + leaves, about `1/N` of tenant addresses move. +- **Placement is per node.** The hash includes the VRF's Linux routing table + ID, which each compute node allocates for itself. Every node builds the same + Maglev table, but the same tenant address on two nodes may land on two + shards. Nothing depends on it doing otherwise: a tenant's NAT sessions, + its pins and the source SID a shard replies to are all specific to the node + it runs on, so a workload that moves to another node starts new sessions + wherever it lands. +- **`source` hashes the tenant's address.** Every connection from one tenant + address uses one shard per class and so one public address per family, + which RFC 4787 REQ-2 asks for and which sites that tie a login to the + client address rely on. One heavy tenant address can never use more than + one shard. `flow` hashes the 5-tuple and spreads that address over every + shard, but gives up the single public address. Placement balances + addresses (or flows), not bytes: shards serving a few heavy tenants carry + more traffic than their count suggests. +- **Fragments.** In `flow` mode every fragment of a datagram, the first + included, hashes on its addresses and protocol only, so the fragments of + one datagram reach one shard. That keeps the shard choice consistent; it + says nothing about whether the shard translates fragments, which the NAT + datapath does not fully support (see Known constraints). +- **Pins hold sessions in place.** The first packet from a tenant address + (in `flow` mode, from a flow) records the shard it went to. Later packets + follow that pin while the shard is reachable, even after a new shard joins + or the pinned shard starts draining. A pin expires after + `GALACTIC_CNI_EGRESS_PIN_IDLE` without a packet, which matches the shard's + longest idle timeout, so a pin never outlives the session it protects. Pins + live in a 65536-entry LRU on each compute node and are not shared between + nodes. One evicted under pressure falls back to the Maglev table, which + gives the same shard unless the membership changed since it was made. `0` + turns pins off. +- **Sessions are not replicated.** A session lives on one shard. When a shard + fails, its sessions break and their tenants are placed elsewhere; only + sessions on surviving shards continue. + +### Draining and removing a shard + +Removing a shard without breaking its sessions takes three steps: + +1. Set `spec.drain: true`: + + ```sh + kubectl patch egressshard -n galactic-system --type=merge -p '{"spec":{"drain":true}}' + ``` + + A draining shard takes no new tenant address, and every pinned one keeps + reaching it until it goes idle. The shard itself changes nothing: it keeps + translating, advertising its SID and receiving replies. +2. Wait for its sessions to end: `galactic_nat_sessions` on its node falls to + zero, or to a level you accept losing. A tenant address that keeps + sending refreshes its pin forever and keeps the shard busy; drain cannot + move it without breaking its sessions, so decide how long to wait. +3. Delete the `EgressShard` (and remove the node's `nat=enabled` label). + +Deleting an `EgressShard` is not a drain. Deletion withdraws the shard's +advertisement and clears its datapath at once, every session on it breaks, +and `galactic-cni` drops it from every group as soon as it sees the deletion. +With pins turned off (`GALACTIC_CNI_EGRESS_PIN_IDLE=0`) there is nothing to +hold a tenant on a draining shard, so draining moves every one of its tenant +addresses at once and breaks their sessions just as a failure would. + +### When nothing is reachable + +There is no spill-over to another metro. When no shard of a class is +reachable, a new attachment fails its ADD, as it does in ordered mode, and an +existing VRF's packets of that class are dropped and counted as `group_empty` +rather than sent untranslated into the node's main table. A cluster with no +`EgressShard` at all gives VRFs no egress route, which is not an error. + +### How the groups are published + +A group is one immutable snapshot: its Maglev table and every member's SID, +next hop, drain state and slot generation in a single value. `galactic-cni` +writes a new one-entry inner map for every change and swaps it into +`egress_shard_groups`, an `ARRAY_OF_MAPS`, in one update. A packet therefore +sees the old snapshot or the new one, never a mix of the two; an earlier +design that updated ordinary array values in place let a packet read one +shard's SID with another's MAC addresses while a slot was rewritten. + +Every member slot carries a generation that identifies the shard's tenure of +it; a pin is honoured only while its slot still carries the generation it +recorded. Generations come from a counter published in the snapshot +(`next_slot_generation`) and a high-water mark in the running process that +also covers snapshots it built but failed to publish. Neither goes down: +disabling a group keeps the counter, and a restarted `galactic-cni` resumes +from the published value. A generation only an unpublished snapshot held was +never visible to a packet, so no pin can hold it. The counter is 32 bits and +skips 0; wrapping takes four billion slot assignments, far longer than any +pin lives. + +### Turning it on and off + +Follow [active-active-rollout.md](active-active-rollout.md). In short: install +the prerequisites, set `GALACTIC_CNI_EGRESS_MODE=hashed` and roll +`galactic-cni`. Within one sweep of starting, each node builds the groups and +moves every VRF route that pointed at a shard onto its class's group. The +move breaks sessions whose new shard differs from the one they were on, +roughly `(N-1)/N` of them, once. + +To go back, set `GALACTIC_CNI_EGRESS_MODE=ordered` and roll `galactic-cni` +again, with `GALACTIC_CNI_EGRESS_SHARD_SIDS` still set on the init container. +Each node first marks its groups closing, so no CNI ADD writes a new route +naming one while they keep forwarding; then moves every route back to the +first reachable shard of the static list; and disables the groups only after +a sweep that read and rewrote every route without an error and found none +left. A sweep that fails leaves the groups forwarding and is retried. A CNI +ADD that read the groups open checks again after writing its routes and +rewrites them in ordered mode if the groups closed meanwhile. +`galactic_cni_egress_group_routes` reaching `0` on every node is the signal +that no route names a group any more. **Do not roll back to a release that +predates hashed mode until it does.** An older `usid_egress` reads a group +route as a redirect to a nonexistent interface and drops the traffic. + +### Observability + +| Metric | Meaning | +| ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | +| `galactic_cni_egress_mode{mode}` | 1 for the configured mode. Always present, so a node configured for hashed mode that never published a group still says so. | +| `galactic_cni_egress_shard_watch_synced` | 1 once the `EgressShard` watch has listed the cluster. | +| `galactic_cni_egress_group_routes{class}` | Routes naming each class's group after the last successful sweep. Must be 0 before rolling back to an older release. | +| `galactic_cni_egress_sweep_errors_total{stage}` | Sweep failures: `apply`, `refresh`, `watch`, `close`, `disable`. | +| `galactic_cni_egress_pool_members{class,state}` | Each class's shards: `active`, `draining`, `unreachable`, and `ineligible` (cannot translate the class). | +| `galactic_cni_egress_pool_enabled{class}` | 1 while the class's group takes new routes; 0 while it is being retired. | +| `galactic_cni_egress_pool_min_active` | `GALACTIC_CNI_EGRESS_POOL_MIN_ACTIVE`. | +| `galactic_cni_egress_shard_packets_total{shard_sid,class}` | Tenant packets this node sent to each shard for each class, since the shard took its slot. `..._bytes_total` is the same in bytes. | +| `galactic_cni_egress_shard_selections_total{result}` | How each packet was placed: `maglev`, `pin_hit`, `pin_stale` (re-placed), or dropped as `group_empty`, `shard_dead`, `parse_failed`. | + +A node that never published a group exports no `pool_*` series. That is not +a healthy empty pool: read it together with `galactic_cni_egress_mode` and +`galactic_cni_egress_shard_watch_synced`. `config/monitoring/` alerts on these +(`GalacticEgressPoolEmpty`, `GalacticEgressPoolBelowMinimum`, +`GalacticEgressClassUnserved`, `GalacticEgressShardGroupDropping`, +`GalacticEgressShardWatchNotSynced`, `GalacticEgressSweepFailing`, +`GalacticCNIMetricsDown`). That file is the tested source of the rules, not +where they run: on Datum's infra the rules are evaluated centrally from +infra's own copy, so they take effect only once infra installs them there. +To compare shards' load from their own side, compare `galactic_nat_sessions` +across the cluster's shard nodes. + ## Session table Every session is two rows in `nat_conn_table`, a 65536-entry LRU hash: a @@ -582,15 +798,30 @@ kubectl get egressshard -n galactic-system -o jsonpath='{.status.conditio Verified against the current working tree as of this writing — worth knowing before you rely on this component in production: -- **Not load-balanced across shards.** `EgressDefaultRouteAdd` - (`internal/plumbing/srv6/egress.go`) installs only the **first - resolvable** SID from `GALACTIC_CNI_EGRESS_SHARD_SIDS` as a tenant VRF's - default egress route — every other configured shard sits as cold - standby, not sharing load. Selection happens only at CNI ADD: a shard - that becomes unreachable later is not replaced on attachments already - made. An earlier version of this mechanism did - spread load across all shards via ECMP; that capability was dropped - during a later datapath migration and has not been reintroduced. +- **Ordered mode is active/standby.** With the default + `GALACTIC_CNI_EGRESS_MODE=ordered`, every tenant VRF uses the first + resolvable SID of `GALACTIC_CNI_EGRESS_SHARD_SIDS`, and every other shard + stands by. `galactic-cni run` re-checks the choice every 30 seconds and + moves a VRF to an earlier shard that becomes reachable, or off one that + stops being reachable, which strands that VRF's sessions. Hashed mode + ([above](#active-active-egress)) spreads tenants across every shard. +- **Hashed mode cannot balance one heavy tenant address** under the default + `source` hash: all of its connections use one shard. A metro with a few + large tenants and N shards balances unevenly. `flow` mode fixes that at + the cost of a single public address per tenant address. +- **Session state is not shared between shards.** A session belongs to one + shard and dies with it. In hashed mode a failure costs only that shard's + sessions. +- **Hashed placement balances addresses, not bytes, and is per node.** A few + heavy tenants make shard load uneven however many addresses each carries, + and the same tenant address on two compute nodes can land on two shards, + since the hash includes each node's own VRF table ID. +- **A busy tenant can keep a draining shard busy indefinitely.** Its pin is + refreshed by every packet, so it never moves while it keeps sending. Pins + are bounded (65536 per compute node) and local to that node. +- **Fragments are not translated end to end.** The shard drops fragmented + IPv4 replies (`nat64_v4_fragment`), and hashed mode's fragment handling only + keeps a datagram's fragments on one shard; it adds no translation support. - **No mechanism announces a shard's public address to the actual internet border.** `Status.ShardAddress` is reachable fabric-wide via BGP/EVPN, but nothing in this repo redistributes it out to a real @@ -665,9 +896,11 @@ knowing before you rely on this component in production: - **The dispatcher outlives the shard.** In dispatch mode the root stays on the uplinks after the shard is removed, passing every packet once the slot's lease lapses. `hack/xdp-dispatch-release.sh` detaches it. -- **The CNI's shard list is a second copy of every shard SID.** - `GALACTIC_CNI_EGRESS_SHARD_SIDS` is set by hand and is not derived from - `EgressShard` status, so the two can disagree. +- **In ordered mode the CNI's shard list is a second copy of every shard + SID.** `GALACTIC_CNI_EGRESS_SHARD_SIDS` is set by hand and is not derived + from `EgressShard` status, so the two can disagree. Hashed mode reads the + `EgressShard`s instead, and uses the list only to move VRFs back when + switching to ordered mode. ## See also