From a29c15b4508ce64cd932fbb4cb716229cfcff0f5 Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Tue, 7 Jul 2026 10:56:05 +0500 Subject: [PATCH 1/5] design-proposal: tenant external network connectivity via a gateway VM Adds a design proposal for tenant-managed site-to-site connectivity (IPsec or WireGuard) terminated in a KubeVirt VM that NAT-bridges an external site to the tenant's managed-app ClusterIPs, both directions, without granting tenants privileged host-cluster pods. Includes the end-to-end prototype validation results and the relationship to the ClusterMesh/Kilo proposal (#7). Co-Authored-By: Claude Opus 4.8 (1M context) Signed-off-by: Myasnikov Daniil --- .../tenant-site-gateway/README.md | 222 ++++++++++++++++++ 1 file changed, 222 insertions(+) create mode 100644 design-proposals/tenant-site-gateway/README.md diff --git a/design-proposals/tenant-site-gateway/README.md b/design-proposals/tenant-site-gateway/README.md new file mode 100644 index 0000000..4cdd041 --- /dev/null +++ b/design-proposals/tenant-site-gateway/README.md @@ -0,0 +1,222 @@ +# Tenant-managed external network connectivity via a gateway VM (IPsec / WireGuard) + +- **Title:** `Tenant-managed external network connectivity via a gateway VM (IPsec / WireGuard)` +- **Author(s):** `@myasnikovdaniil` +- **Date:** `2026-07-07` +- **Status:** Draft + +## Overview + +Cozystack tenants need site-to-site VPN connectivity between their workloads and external networks, in both directions: exposing a tenant-managed application to a remote site (inbound), and letting a tenant-managed application reach a service that lives only behind a tunnel (outbound). This must work for any application — databases, message queues, object storage, gRPC, raw UDP — not one special case. + +This proposal terminates the tunnel (IPsec or WireGuard) **inside a KubeVirt VM** rather than in a pod. The privileged dataplane (`NET_ADMIN`, XFRM / a `wireguard` interface, `ip_forward`) lives in a guest kernel isolated by hardware virtualization, so its blast radius is contained by the VM boundary — the host node and other tenants are unaffected. The gateway VM is a normal member of the tenant's pod network, so it reaches any tenant `ClusterIP` Service; it performs all address translation (DNAT/SNAT), and the managed applications are never modified and never speak the VPN — they only ever see ordinary `ClusterIP`s on their own network. We propose shipping this as a Cozystack catalog app, provisionally `packages/apps/site-gateway`. + +The design has been prototyped and validated end-to-end (see [Testing](#testing)). + +## Scope and related proposals + +- **Related — [`cross-cluster-tenant-mesh` (ClusterMesh / Kilo)](https://github.com/cozystack/community/pull/7).** That proposal connects cooperative Kubernetes clusters into a routed WireGuard node-to-node mesh for high-throughput cases (its motivating example is a tenant cluster consuming host Ceph). This proposal is complementary and occupies a different niche — an arbitrary external site (not a Kubernetes cluster), NAT-bridged to managed apps that live in the tenant namespace. See [Alternatives considered](#alternatives-considered) for a side-by-side and a "why two mechanisms" discussion. Notably, PR #7 explicitly defers the "tenant ↔ tenant-namespace applications" and "NAT-egress" integration — precisely the space this proposal fills. +- **Deferred here:** a platform-run (non-VM) managed variant; HTTP/L7 ingress (the existing tenant Ingress / Gateway API already covers public HTTP); preservation of the original client source IP inbound. + +## Context + +Cozystack runs tenant workloads under a hardened networking model: + +- **Namespace hardening.** Cozystack applies Pod Security Standards to tenant namespaces. The platform operator only sets `pod-security.kubernetes.io/enforce=privileged` on a namespace when a platform-declared package requests it. A regular tenant cannot schedule a privileged, `NET_ADMIN`, or `hostNetwork` pod — deliberately, because such a pod executes against the node kernel and could manipulate host networking or observe other tenants' traffic. +- **Default pod network vs. custom VPC.** Managed apps (databases and friends) run as ordinary pods on the **default cluster pod network** and are reached via `ClusterIP` Services + CoreDNS. They are not placed on a custom kube-ovn VPC; VPCs are isolated from the default network and the managed-app operators reconcile over the default network — so a managed app cannot simply be "moved onto a VPC." +- **Dual-homed VMs.** KubeVirt VMs are the exception: every VM always has a `default` pod-network NIC (and can carry additional multus interfaces). A VM is therefore a first-class member of the default pod network. This is the primitive the design leans on. + +### The problem + +A tenant today has no supported way to: + +- **Inbound:** let a remote site reach a tenant-managed app (a Postgres instance, an internal API, an object store) over a tunnel, without exposing it to the public internet. +- **Outbound:** let a tenant app call a service (a legacy database, a licensing server, an on-prem API) that is only routable through a tunnel. + +The naive approaches all require either a privileged host-cluster pod (a tenant-escape vector — defeats the hardening above) or changes to shared, cluster-wide network infrastructure (cross-tenant collision risk). We want a self-service, tenant-scoped mechanism that generalizes to any protocol and touches neither the host kernel nor the managed apps. + +## Goals + +- Give tenants **self-service** site-to-site connectivity, both directions, through a catalog app. +- Support **inbound** exposure of any tenant-managed app to a remote site over the tunnel. +- Support **outbound** connections from any tenant app to services reachable only via the tunnel. +- Be **app-agnostic** — work for any L4 protocol (TCP/UDP, any port) without modifying the target apps. +- Keep the privileged dataplane **contained** in a VM guest; add no privileges to the host cluster and none to managed apps. +- Keep all bridging/NAT **tenant-scoped**; make no changes to shared cluster routers. +- Support **both IPsec and WireGuard** as tunnel backends (interop vs. simplicity — see [Design](#design)). + +### Non-goals + +- **HTTP/L7 ingress.** Public HTTP(S) is served by the existing tenant Ingress / Gateway API; the gateway VM is for site-to-site VPN and non-HTTP protocols. +- **Preserving the original client source IP inbound.** SNAT hides it (see [Failure and edge cases](#failure-and-edge-cases)); source-IP ACLs are out of scope for the first iteration. +- **A platform-managed, always-on VPN service.** This is a tenant-deployed catalog app; a platform-run variant is possible future work. +- **Routing a remote CIDR onto the shared default-VPC router** — explicitly rejected (cross-tenant collisions). + +## Design + +### Principle + +Terminate the tunnel in a KubeVirt VM. The VM guest kernel isolates `NET_ADMIN`/XFRM from the host, so the tenant effectively runs a privileged network appliance without the platform ever granting a privileged pod. The VM is dual-homed on the tenant's default pod network and is the sole holder of the tunnel; it DNAT/SNATs between the tunnel and the tenants' managed-app `ClusterIP`s. + +### Topology + +```mermaid +flowchart TB + ext["External site
(on-prem router / cloud VPN / another org)"] + lb["LoadBalancer VIP
(tunnel UDP ports)"] + subgraph node["Worker node"] + gw["Gateway VM (VyOS)
tunnel termination + DNAT/SNAT + firewall + MSS clamp"] + a["managed app A
(ClusterIP)"] + b["managed app B
(ClusterIP)"] + c["managed app C
(ClusterIP)"] + end + ext -->|"IPsec ESP-in-UDP / WireGuard UDP"| lb --> gw + gw --> a & b & c +``` + +Everything privileged lives inside the VM guest. Managed apps only ever see `ClusterIP`s on their own default network. + +### Inbound path (external site → any tenant app) + +The remote peer targets a virtual "service-exposure" address that the gateway owns and maps to an app `ClusterIP`. + +1. Remote peer sends tunnel traffic to the gateway's public VIP (a `LoadBalancer` Service on the tunnel's UDP ports). +2. The VIP delivers to the gateway VM's pod NIC; the VM decrypts the tunnel. +3. **DNAT**: the virtual destination (e.g. `10.200.0.10:5432`) → the target app's `ClusterIP:port`. One entry per exposed app — the exposure table. +4. **SNAT (masquerade)**: source → the gateway's own pod IP. Mandatory: the cluster has no route back to the remote peer's inner IP, so without SNAT the app's reply would leave via the default route and be black-holed. +5. The VM forwards onto the pod network; the node resolves the `ClusterIP` to a backend pod; the app sees the connection with the gateway pod IP as client. Replies retrace the path and are re-encrypted back over the tunnel. + +### Outbound path (any tenant app → remote service over tunnel) + +The tenant app never routes to the remote address directly; it talks to a **local Service** whose endpoint is the gateway VM. + +1. The app connects to a local Service name the chart creates (e.g. `remote-db..svc:port`) — the only app-side change is the target hostname. +2. The node routes the `ClusterIP` to the gateway VM. +3. **DNAT**: the local listener → the real remote `IP:port` behind the tunnel; the VM encrypts and sends it over the tunnel; SNAT into the tunnel's inner subnet lets replies return. + +**Rejected alternative — a static route on the shared default-VPC router.** Injecting a tenant's remote RFC1918 CIDR into the cluster-wide default router risks cross-tenant address collisions and leakage. The local-Service approach keeps every remote address strictly inside the tenant's gateway VM. + +### Why it generalizes to all apps + +- The app only ever sees a `ClusterIP` / Service on its own network; it speaks no VPN and needs no route to the far side. +- DNAT is **L4** (TCP and UDP, any port), not L7 — so it works for databases, queues, object storage, gRPC, and raw UDP, unlike an HTTP-only path. +- The gateway VM is the only privileged component, and it is a contained VM guest. + +### Tunnel backend: IPsec or WireGuard + +VyOS terminates both natively and the DNAT/SNAT bridging is identical — only the transport differs. `tunnel.type` selects the backend per peer. + +- **WireGuard — the simpler default on an overlay CNI.** WireGuard is UDP-native (a single configurable UDP port), so it rides the pod overlay as ordinary UDP: no ESP, no NAT-T, no forced encapsulation. It is simpler (static keypairs, no IKE) and has smaller header overhead. Preferred for tenant↔tenant and tenant↔Cozystack links. (See the validated finding in [Testing](#testing) that makes this concrete.) +- **IPsec — for interop.** Much external/enterprise gear only speaks IKEv2/IPsec; when the remote site is not under the tenant's control this is often the only option. Supported, with the encapsulation requirement described in [Testing](#testing). + +### Build options + +- **VyOS appliance (recommended).** A single network OS does IPsec, WireGuard, native DNAT/SNAT, and firewalling; no separate proxy needed; configured declaratively via cloud-init. +- **Alpine + libreswan/strongSwan (or wireguard-tools) + haproxy.** A leaner, self-composed image if a network-OS dependency is undesirable; more moving parts. + +## User-facing changes + +A new catalog app, provisionally **`packages/apps/site-gateway`** (VM-backed), appears in the tenant dashboard. The tenant fills a values form; the platform-authored chart renders the gateway VM, its tunnel + NAT + firewall config, the local Services for outbound targets, and a `LoadBalancer` Service for the tunnel's UDP ports. Sketch of the values schema (cozyvalues-gen annotated): + +```yaml +## @param tunnel.type {string} Backend: "wireguard" (default; UDP-native) or "ipsec". +tunnel: + type: wireguard + +## @param peer.address {string} Public address of the remote peer / WireGuard endpoint. +peer: + address: "" + ## backend-specific auth: IPsec IKE/PSK-or-cert; WireGuard peer public key + allowedIPs. + auth: + secretRef: "" + +## @param exposedServices {array} Inbound: {name, listenPort, targetService, port} +exposedServices: [] + +## @param remoteTargets {array} Outbound: {name, localPort, remoteHost, remotePort} +remoteTargets: [] + +## @param resources {object} VM sizing (cpu/memory). +resources: + cpu: "1" + memory: "1Gi" +``` + +No existing app, CRD, or API changes. + +## Upgrade and rollback compatibility + +Purely additive and opt-in. No migration; existing clusters, manifests, and APIs are unaffected. The gateway VM image is a new artifact built and published by the platform. Rollback is deletion of the app instance (its `VMInstance`/`VMDisk` and the generated Services), which removes the gateway entirely; nothing else in the tenant is touched. + +## Security + +- **Contained privilege.** The privileged dataplane lives in a VM guest kernel isolated by hardware virtualization; a compromise or misconfiguration inside the gateway cannot manipulate the host node's networking or observe other tenants. The platform never hands a tenant a privileged host-cluster pod. +- **Managed apps untouched.** No new privileges or config on the apps; they stay ordinary pods on the default network. +- **Tenant-scoped bridging.** All translation is NAT inside the VM; nothing changes on the shared cluster router, so there is no cross-tenant surface. +- **Firewall allow-list.** The gateway restricts which tunnel-side sources may reach which exposed ports — exposure is explicit, not "the whole tunnel reaches everything." +- **Tenant-supplied secrets.** PSK / certificate / WireGuard key material is provided via a Secret reference and mounted into the guest; the chart must avoid persisting it in plaintext cloud-init user-data at rest (open question). + +## Failure and edge cases + +- **Missing SNAT → inbound reply black-holed.** Without the source-NAT rule the app replies to an unroutable tunnel address and the flow times out. SNAT is mandatory (validated). +- **MTU / double encapsulation.** The overlay already lowers MTU and the tunnel adds overhead; without MSS clamping (and/or a lowered tunnel MTU) large TCP packets black-hole. The gateway sets an MSS clamp by default. +- **Native ESP dropped by the CNI overlay (IPsec).** On Cilium/kube-ovn, native ESP (IP proto 50) does not traverse the overlay even pod-to-pod; ESP-in-UDP (forced UDP encapsulation) is required unconditionally (validated — see Testing). WireGuard, being UDP-native, is unaffected. +- **DNAT must target stable `ClusterIP`s**, never ephemeral pod IPs. +- **Source IP is lost inbound** (SNAT) — only relevant for apps with source-IP ACLs. +- **VM image must be bootable and 512-native.** DRBD-backed (4K-sector) StorageClasses cannot boot a 512-native GPT image; the disk must sit on a 512-native StorageClass or use a KubeVirt `blockSize` override. The image must ship a real bootloader (validated the hard way — see Testing). +- **Single gateway is a per-tenant SPOF** until HA is added (open question). + +## Testing + +The design was **prototyped and validated end-to-end** on a development Cozystack cluster (KubeVirt, Cilium + kube-ovn, LINSTOR), using two gateway VMs — one acting as the tenant gateway, one simulating the external site — with a real tenant-managed Postgres as the inbound target. IPsec was validated first; WireGuard is the next backend to validate (open question). All items passed: + +| # | Item | Result | +|---|------|--------| +| 1 | IKEv2 SA establishes both sides | PASS | +| 2 | Inbound: external site → virtual VIP → DNAT/SNAT → managed Postgres (real server response) | PASS | +| 3 | Outbound: gateway client → tunnel → remote listener | PASS | +| 4 | SNAT-required negative test (removing SNAT black-holes the reply) | PASS | +| 5 | MTU / MSS clamp (working tunnel MTU 1320, clamp 1280; multi-MB transfer, no stall) | PASS | + +**Key finding — native ESP is dropped by the CNI overlay.** IKE (UDP) negotiated and the SA came up, but the ESP data plane was silently dropped even pod-to-pod (receiver saw zero ESP, 100% loss). Forcing ESP-in-UDP encapsulation on both peers restored bidirectional traffic. Consequence: on an overlay CNI the IPsec backend must always force UDP encapsulation — and **WireGuard, being UDP-native, sidesteps this entirely**, which is a strong argument for it as the default backend. + +Planned automated coverage before implementation lands: helm-unittest for chart rendering across backends; an e2e that stands up the two-VM topology and asserts the inbound/outbound flows and the SNAT-required and MSS behaviors. + +## Rollout + +- **Phase 1 — IPsec backend** (prototype validated): ship `site-gateway` with the IPsec backend (forced UDP encapsulation) + the inbound/outbound NAT machinery. +- **Phase 2 — WireGuard backend**: validate WireGuard over the overlay and, if confirmed simpler/robust, make it the default `tunnel.type`. +- **Phase 3 — hardening**: HA (active/standby, shared VIP), tunnel-state observability in the dashboard, MSS/MTU auto-tuning. + +## Open questions + +- **Secret handling.** Best shape for mounting PSK/cert/WireGuard-key material into the guest without persisting it in plaintext cloud-init at rest. +- **VIP allocation.** One LoadBalancer VIP per gateway vs. sharing; interaction with the tenant's LB address pool and quotas. +- **HA.** Is active/standby (e.g. VRRP + shared VIP) in scope for a later iteration? A single VM is a per-tenant SPOF. +- **Observability.** Surfacing tunnel state (SA up/down, rekey, counters) through existing dashboards. +- **MTU auto-tuning.** Derive the MSS clamp / tunnel MTU from the detected overlay MTU, or require an explicit value. +- **WireGuard backend.** Validate the handshake/SA over the overlay and confirm identical DNAT/SNAT bridging before defaulting to it. +- **Relationship to ClusterMesh (PR #7).** Whether to present this and ClusterMesh as one coordinated "tenant connectivity" story, and where the boundary sits. + +## Alternatives considered + +- **Give tenants privileged / `NET_ADMIN` host pods.** Rejected — tenant escape; defeats namespace hardening. The whole design exists to avoid this. +- **Route the remote CIDR into the shared default-VPC router.** Rejected — the router is shared cluster-wide; injecting tenant RFC1918 CIDRs causes cross-tenant collisions and leakage. +- **Put managed apps on a custom kube-ovn VPC.** Not viable — VPCs are isolated from the default network and managed-app operators reconcile over the default network; only VMs are dual-homed. +- **Platform-run kube-ovn VPC-NAT-Gateway with the tunnel.** Possible future managed variant, but more platform work and it couples XFRM to the host kernel + OVS. The VM approach contains the privilege more cleanly and ships sooner. +- **Traefik / L7 proxy.** Limited — HTTP-only; not general across L4 protocols. + +### ClusterMesh / Kilo (PR #7), and why two mechanisms + +[PR #7](https://github.com/cozystack/community/pull/7) pursues an overlapping goal (tenant access to services across a boundary) but is a different shape, and neither mechanism subsumes the other: + +| Axis | ClusterMesh / Kilo (PR #7) | This proposal (gateway VM) | +|------|---------------------------|----------------------------| +| Remote end | A cooperative Kubernetes cluster running Kilo, reached via a kubeconfig | An arbitrary external site — not Kubernetes, no Kilo, no kubeconfig | +| Tenant workload served | The tenant's managed Kubernetes cluster (guest-VM pods) | Managed apps in the host-cluster tenant namespace (ClusterIPs) | +| Transport | WireGuard node-to-node mesh (`mesh-granularity=cross`) | IPsec or WireGuard site-to-site, one VM | +| Address plane | Routed pod-CIDRs, no NAT; disjoint CIDRs required | NAT (DNAT/SNAT) to ClusterIPs; tolerates overlap | +| Throughput | Full node×node mesh — direct pod-IP to many backends (built for Ceph) | Single gateway — app-level flows, not Ceph-scale | +| Privilege | Kilo node `NET_ADMIN`, on the tenant side inside guest-cluster VMs → contained | `NET_ADMIN` inside the gateway VM guest → contained | + +**Kilo cannot terminate a third-party VPN** (both ends must run Kilo with a kubeconfig), and **a single gateway VM cannot serve the Ceph-scale mesh** (throughput/topology incompatible with funneling). They are different layers: a platform storage fabric vs. a tenant-edge VPN concentrator — analogous to a cloud provider shipping both VPC Peering and a VPN Gateway. Both share the principle that matters here: no privileged pods in the shared host namespace; privilege is contained in guest-cluster VMs (Kilo) or a dedicated gateway VM (this proposal). Recommended framing: lead with the gateway VM as the single tenant-facing primitive (covers external sites, tenant↔tenant, and managed-app exposure), and use ClusterMesh only for the high-throughput cross-cluster storage case. From 72db47ee62705a39492f9acde6fe0b92a1fe5110 Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Tue, 7 Jul 2026 17:36:58 +0500 Subject: [PATCH 2/5] design-proposal: add HA section and address review feedback Adds a High availability section (KubeVirt live-migration for planned maintenance; Service-fronted active/passive and kube-ovn allowed-address-pairs shared-VIP for unplanned failure), with the shared-VIP mechanism validated on a development cluster. Attributes the pod-to-pod drop of non-TCP/UDP/ICMP/SCTP IP protocols (VRRP proto 112, ESP proto 50) to Cilium's conntrack rather than the geneve tunnel, unifying it with the native-ESP finding. Also addresses review feedback: per-target-unique outbound listener ports, a concrete blockSize example, a note that SNAT keeps the gateway anti-spoofing clean, and a stronger secret-handling open question. Co-Authored-By: Claude Opus 4.8 (1M context) Signed-off-by: Myasnikov Daniil --- .../tenant-site-gateway/README.md | 32 ++++++++++++++----- 1 file changed, 24 insertions(+), 8 deletions(-) diff --git a/design-proposals/tenant-site-gateway/README.md b/design-proposals/tenant-site-gateway/README.md index 4cdd041..85878c5 100644 --- a/design-proposals/tenant-site-gateway/README.md +++ b/design-proposals/tenant-site-gateway/README.md @@ -91,8 +91,8 @@ The remote peer targets a virtual "service-exposure" address that the gateway ow The tenant app never routes to the remote address directly; it talks to a **local Service** whose endpoint is the gateway VM. 1. The app connects to a local Service name the chart creates (e.g. `remote-db..svc:port`) — the only app-side change is the target hostname. -2. The node routes the `ClusterIP` to the gateway VM. -3. **DNAT**: the local listener → the real remote `IP:port` behind the tunnel; the VM encrypts and sends it over the tunnel; SNAT into the tunnel's inner subnet lets replies return. +2. The node routes the `ClusterIP` to the gateway VM on a **per-target-unique listener port** (the local Service still exposes the standard port, e.g. 5432, mapped to a unique port on the VM). The uniqueness is required so the VM can tell apart multiple remote targets that share the same destination port. +3. **DNAT**: that unique local listener port → the real remote `IP:port` behind the tunnel; the VM encrypts and sends it over the tunnel; SNAT into the tunnel's inner subnet lets replies return. **Rejected alternative — a static route on the shared default-VPC router.** Injecting a tenant's remote RFC1918 CIDR into the cluster-wide default router risks cross-tenant address collisions and leakage. The local-Service approach keeps every remote address strictly inside the tenant's gateway VM. @@ -114,6 +114,20 @@ VyOS terminates both natively and the DNAT/SNAT bridging is identical — only t - **VyOS appliance (recommended).** A single network OS does IPsec, WireGuard, native DNAT/SNAT, and firewalling; no separate proxy needed; configured declaratively via cloud-init. - **Alpine + libreswan/strongSwan (or wireguard-tools) + haproxy.** A leaner, self-composed image if a network-OS dependency is undesirable; more moving parts. +### High availability + +HA splits by failure mode: + +- **Planned maintenance (node drain):** KubeVirt **live-migration** relocates the gateway VM with its state intact (conntrack, tunnel SA, pod IP preserved) — no tunnel drop. Preferred for maintenance; needs no standby. (Requires migratable storage — see the storage caveat below.) +- **Unplanned node/VM failure:** needs a standby plus a failover trigger. Two mechanisms, both examined on kube-ovn (see [Testing](#testing) for the shared-VIP validation): + + 1. **Service-fronted active/passive.** The external `LoadBalancer` (tunnel) and internal `ClusterIP` (outbound) Services select whichever gateway pod is active; failover = re-pointing endpoints when the active goes unready. The stable address is the Service VIP (CNI-managed) — no floating L2 VIP, so no port-security interaction at all. Cost: needs a small leader-election/lease agent (or controller) to flip the active endpoint, and failover takes seconds (endpoint reconvergence). CNI-agnostic. + 2. **Shared VIP via kube-ovn allowed-address-pairs (AAP).** A kube-ovn `Vip` plus the `ovn.kubernetes.io/aaps` pod annotation adds *only* the VIP to the port's OVN `port_security` (which stays enforced) — validated: the VIP is reachable, moves between the two gateway pods through OVN on owner change, and any other source address is still dropped, so anti-spoofing is **scoped rather than disabled** and the VM stays fully tenant-controlled (no need to withhold direct VM/cloud-init access). **Caveat:** VRRP advertisements (IP proto 112) do not reach the peer pod, so keepalived split-brains and cannot elect a master over the pod network. Root cause (independently verified): the CNI drops *all* non-TCP/UDP/ICMP/SCTP IP protocols pod-to-pod — in policy-enforced tenant pods this is **Cilium's conntrack** (`bpf_lxc.c`, "CT: Unknown L4 protocol"), and without a policy it is the OVS datapath; it is **not** geneve/inter-node-specific (same-node pods drop it too). This is the same root cause that makes native ESP need UDP encapsulation. To use this path the election must therefore run over a side-channel (e.g. the gateway pair's own tunnel link) or be driven by a controller; the AAP VIP move itself is proven. Also the `aaps` annotation must be baked into the KubeVirt VM's pod template (the current `VMInstance` chart does not expose it), and the VIP rides the pod's real MAC (keep the VyOS default — no virtual-MAC). + +**Recommendation:** iteration 1 ships a single gateway VM with live-migration for maintenance, and treats automatic unplanned-failure HA as a follow-up — neither mechanism is a clean drop-in today (Service-based needs a controller; AAP needs the VRRP-advert-delivery issue resolved). + +**Storage caveat:** live-migration and single-VM reschedule need migratable/replicated storage — but replicated (DRBD) StorageClasses expose 4K sectors, so the image needs the `blockSize` override to boot (see [Failure and edge cases](#failure-and-edge-cases)). A two-VM active/passive pair can instead use node-local 512-native disks. + ## User-facing changes A new catalog app, provisionally **`packages/apps/site-gateway`** (VM-backed), appears in the tenant dashboard. The tenant fills a values form; the platform-authored chart renders the gateway VM, its tunnel + NAT + firewall config, the local Services for outbound targets, and a `LoadBalancer` Service for the tunnel's UDP ports. Sketch of the values schema (cozyvalues-gen annotated): @@ -158,12 +172,12 @@ Purely additive and opt-in. No migration; existing clusters, manifests, and APIs ## Failure and edge cases -- **Missing SNAT → inbound reply black-holed.** Without the source-NAT rule the app replies to an unroutable tunnel address and the flow times out. SNAT is mandatory (validated). +- **Missing SNAT → inbound reply black-holed.** Without the source-NAT rule the app replies to an unroutable tunnel address and the flow times out. SNAT is mandatory (validated). It also keeps the gateway **anti-spoofing-clean**: every packet it emits onto the pod network carries its own pod IP, so the CNI's port-security never sees a foreign source — the dataplane needs no port-security relaxation (that only arises for shared-VIP HA; see High availability). - **MTU / double encapsulation.** The overlay already lowers MTU and the tunnel adds overhead; without MSS clamping (and/or a lowered tunnel MTU) large TCP packets black-hole. The gateway sets an MSS clamp by default. -- **Native ESP dropped by the CNI overlay (IPsec).** On Cilium/kube-ovn, native ESP (IP proto 50) does not traverse the overlay even pod-to-pod; ESP-in-UDP (forced UDP encapsulation) is required unconditionally (validated — see Testing). WireGuard, being UDP-native, is unaffected. +- **Native ESP dropped pod-to-pod (IPsec).** The CNI drops non-TCP/UDP/ICMP/SCTP IP protocols between pods — native ESP (proto 50) included — so ESP-in-UDP (forced UDP encapsulation) is required unconditionally (validated). In policy-enforced tenant pods the dropper is Cilium's conntrack ("Unknown L4 protocol"); it is not geneve-specific (same root cause as the VRRP/proto-112 case — see Testing). WireGuard, being UDP-native, is unaffected. - **DNAT must target stable `ClusterIP`s**, never ephemeral pod IPs. - **Source IP is lost inbound** (SNAT) — only relevant for apps with source-IP ACLs. -- **VM image must be bootable and 512-native.** DRBD-backed (4K-sector) StorageClasses cannot boot a 512-native GPT image; the disk must sit on a 512-native StorageClass or use a KubeVirt `blockSize` override. The image must ship a real bootloader (validated the hard way — see Testing). +- **VM image must be bootable and 512-native.** DRBD-backed (4K-sector) StorageClasses cannot boot a 512-native GPT image; the disk must sit on a 512-native StorageClass or use a KubeVirt `blockSize` override (e.g. `blockSize.custom.logical: 512` / `physical: 4096` on the disk spec). The image must ship a real bootloader (validated the hard way — see Testing). - **Single gateway is a per-tenant SPOF** until HA is added (open question). ## Testing @@ -180,19 +194,21 @@ The design was **prototyped and validated end-to-end** on a development Cozystac **Key finding — native ESP is dropped by the CNI overlay.** IKE (UDP) negotiated and the SA came up, but the ESP data plane was silently dropped even pod-to-pod (receiver saw zero ESP, 100% loss). Forcing ESP-in-UDP encapsulation on both peers restored bidirectional traffic. Consequence: on an overlay CNI the IPsec backend must always force UDP encapsulation — and **WireGuard, being UDP-native, sidesteps this entirely**, which is a strong argument for it as the default backend. +**HA mechanism validation (kube-ovn AAP).** Separately validated (kube-ovn v1.15.10): a `Vip` + `ovn.kubernetes.io/aaps` pod annotation shares a VIP across the two gateway VM pods with `port_security` kept **on**. Proven: (1) without AAP the VIP is dropped (GARP/ping blocked); (2) with AAP the VIP is reachable and lands on the active pod; (3) forcing the active down moves the VIP to the standby through OVN; (4) a bogus source address from the VM is still dropped — anti-spoofing is scoped, not disabled. **Not** working: VRRP advertisements (IP proto 112) do not reach the peer pod, so keepalived cannot elect over the pod network — the election needs a side-channel or controller (see [High availability](#high-availability)). Independently re-verified with a minimal repro: the CNI drops *all* non-TCP/UDP/ICMP/SCTP IP protocols (112, 50, 47, 4) pod-to-pod, both same-node and cross-node — so this is **not** the geneve tunnel. In policy-enforced tenant pods the dropper is **Cilium's conntrack** (`bpf_lxc.c`, "CT: Unknown L4 protocol"); without a policy it is the OVS datapath. This is the same root cause as the native-ESP drop above. An upstream issue, if pursued, targets Cilium (a known conntrack limitation); the OVS-side drop is not yet root-caused. + Planned automated coverage before implementation lands: helm-unittest for chart rendering across backends; an e2e that stands up the two-VM topology and asserts the inbound/outbound flows and the SNAT-required and MSS behaviors. ## Rollout - **Phase 1 — IPsec backend** (prototype validated): ship `site-gateway` with the IPsec backend (forced UDP encapsulation) + the inbound/outbound NAT machinery. - **Phase 2 — WireGuard backend**: validate WireGuard over the overlay and, if confirmed simpler/robust, make it the default `tunnel.type`. -- **Phase 3 — hardening**: HA (active/standby, shared VIP), tunnel-state observability in the dashboard, MSS/MTU auto-tuning. +- **Phase 3 — hardening**: HA (see [High availability](#high-availability) — Service-based or AAP shared-VIP), tunnel-state observability in the dashboard, MSS/MTU auto-tuning. ## Open questions -- **Secret handling.** Best shape for mounting PSK/cert/WireGuard-key material into the guest without persisting it in plaintext cloud-init at rest. +- **Secret handling.** Deliver PSK/cert/WireGuard-key material to the guest without persisting it in plaintext cloud-init at rest — candidate: mount the referenced `Secret` as a separate config disk (or fetch it on first boot via a small init) instead of inlining it in the VM spec; the post-reconciliation read-access boundary needs to be defined. - **VIP allocation.** One LoadBalancer VIP per gateway vs. sharing; interaction with the tenant's LB address pool and quotas. -- **HA.** Is active/standby (e.g. VRRP + shared VIP) in scope for a later iteration? A single VM is a per-tenant SPOF. +- **HA mechanism.** Given VRRP advertisements do not cross the kube-ovn overlay (see Testing), which failover trigger to standardize on — Service/endpoint-based (needs a controller/lease) or AAP shared-VIP with election over a side-channel — and baking the `aaps` annotation into the `VMInstance` chart's pod template. - **Observability.** Surfacing tunnel state (SA up/down, rekey, counters) through existing dashboards. - **MTU auto-tuning.** Derive the MSS clamp / tunnel MTU from the detected overlay MTU, or require an explicit value. - **WireGuard backend.** Validate the handshake/SA over the overlay and confirm identical DNAT/SNAT bridging before defaulting to it. From 4672a2ddda23ce833877304c4d777406e760c16a Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Fri, 10 Jul 2026 18:08:01 +0500 Subject: [PATCH 3/5] design-proposal: restructure into site-router + site-gateway (routed + NAT) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Split the tenant site-to-site connectivity proposal into two co-equal catalog apps over a shared foundation — site-router (routed, source-IP preserving, kube-ovn) and site-gateway (NAT, CNI-agnostic) — instead of a single NAT gateway with routed as a fallback, and rename the proposal to tenant-site-connectivity. Adds day-2 configuration (VyOS HTTPS API driven by a controller), a low-level-entities/controller section, and publishes the tunnel endpoint via the structured external-exposure primitives (#29) instead of a bespoke LoadBalancer. Tightens secret handling: the chart templates the Secret from values (no tenant-created Secret), the WireGuard key is autogenerated in-chart, and the IPsec PSK is the only genuinely tenant-supplied secret. Reorders rollout to lead with routed + MSS clamping + tunnel observability. Co-Authored-By: Claude Opus 4.8 (1M context) Signed-off-by: Myasnikov Daniil --- .../tenant-site-connectivity/README.md | 251 ++++++++++++++++++ .../tenant-site-gateway/README.md | 238 ----------------- 2 files changed, 251 insertions(+), 238 deletions(-) create mode 100644 design-proposals/tenant-site-connectivity/README.md delete mode 100644 design-proposals/tenant-site-gateway/README.md diff --git a/design-proposals/tenant-site-connectivity/README.md b/design-proposals/tenant-site-connectivity/README.md new file mode 100644 index 0000000..9989739 --- /dev/null +++ b/design-proposals/tenant-site-connectivity/README.md @@ -0,0 +1,251 @@ +# Tenant-managed site-to-site connectivity via gateway VMs (`site-router` and `site-gateway`) + +- **Title:** `Tenant-managed site-to-site connectivity via gateway VMs (site-router and site-gateway)` +- **Author(s):** `@myasnikovdaniil` +- **Date:** `2026-07-08` +- **Status:** Draft + +## Overview + +Cozystack tenants need site-to-site connectivity between their workloads and external networks, in both directions — exposing a tenant-managed application to a remote site (inbound), and letting a tenant-managed application reach a service that lives only behind a tunnel (outbound) — and it must work for any application (databases, message queues, object storage, gRPC, raw UDP), not one special case. + +The privileged network dataplane (`NET_ADMIN`, XFRM / a `wireguard` interface, `ip_forward`) is terminated **inside a KubeVirt VM**, not a pod: the guest kernel is isolated by hardware virtualization, so the blast radius is contained by the VM boundary and the platform never grants a tenant a privileged host-cluster pod. The gateway VM is a normal, dual-homed member of the tenant's pod network. + +There are two legitimate ways to bridge such a tunnel to the tenant's workloads, with genuinely different trade-offs, and both are wanted in practice — so this proposal ships them as **two co-equal catalog apps over a shared foundation**: + +- **`site-router`** — *routed* mode. Plain L3 forwarding between the tunnel and the tenant network; the original client source IP is preserved; whole-subnet reachability and ICMP work. Uses kube-ovn routing (kube-ovn-specific). +- **`site-gateway`** — *NAT* mode. The VM DNAT/SNAT-bridges the tunnel to managed-app `ClusterIP`s; CNI-agnostic and fully contained; per-target host:port reachability; the original source IP is not preserved. + +Neither is a fallback for the other — a cluster admin can offer either or both. The design has been prototyped and validated end-to-end (see [Testing](#testing)). (Naming note: these are distinct from the existing `packages/apps/vpn`, which is a client / remote-access VPN — the two apps here are site-to-site network edges.) + +## Scope and related proposals + +- **[#29 structured external exposure](https://github.com/cozystack/community/pull/29).** The tunnel's own external entry point (the UDP listener the remote peer dials) is published through the structured `ServiceExposure` / `ExposureClass` primitives from #29 rather than a bespoke `LoadBalancer` path; that exposure class must support **UDP** (IKE/NAT-T 4500, WireGuard). +- **[#7 ClusterMesh / Kilo](https://github.com/cozystack/community/pull/7).** Complementary and a different niche — a routed WireGuard node-to-node mesh between cooperative Kubernetes clusters. See [Alternatives considered](#alternatives-considered). +- **Deferred here:** a platform-run (non-VM) managed variant; HTTP/L7 ingress (the existing tenant Ingress / Gateway API covers public HTTP); a per-tenant egress IP (future work — the gateway VM is its natural home later). + +## Context + +Cozystack runs tenant workloads under a hardened networking model: + +- **Namespace hardening.** Cozystack applies Pod Security Standards to tenant namespaces. The platform operator only sets `pod-security.kubernetes.io/enforce=privileged` on a namespace when a platform-declared package requests it. A regular tenant cannot schedule a privileged, `NET_ADMIN`, or `hostNetwork` pod — deliberately, because such a pod executes against the node kernel and could manipulate host networking or observe other tenants' traffic. +- **Default pod network vs. custom VPC.** Managed apps (databases and friends) run as ordinary pods on the **default cluster pod network** and are reached via `ClusterIP` Services + CoreDNS. They are not placed on a custom kube-ovn VPC; VPCs are isolated from the default network and the managed-app operators reconcile over the default network — so a managed app cannot simply be "moved onto a VPC." +- **Dual-homed VMs.** KubeVirt VMs are the exception: every VM always has a `default` pod-network NIC (and can carry additional multus interfaces). A VM is therefore a first-class member of the default pod network. This is the primitive the design leans on. + +### The problem + +A tenant today has no supported way to let a remote site reach a tenant-managed app over a tunnel without exposing it to the public internet (inbound), nor to let a tenant app call a service that is only routable through a tunnel (outbound). The naive approaches all require either a privileged host-cluster pod (a tenant-escape vector — defeats the hardening above) or changes to shared, cluster-wide network infrastructure (cross-tenant collision risk). We want a self-service, tenant-scoped mechanism that generalizes to any protocol and touches neither the host kernel nor the managed apps. + +## Goals + +- Give tenants **self-service** site-to-site connectivity, both directions, through catalog apps. +- Support **inbound** exposure of any tenant-managed app and **outbound** reach to tunnel-only services, for any L4 protocol, without modifying the target apps. +- Keep the privileged dataplane **contained** in a VM guest; add no privileges to the host cluster and none to managed apps. +- Offer **both** a routed mode (source-IP-preserving, L3) and a NAT mode (contained, CNI-agnostic) as **co-equal, admin-gateable** choices — not one as a fallback of the other. +- Support **both IPsec and WireGuard** as tunnel backends (interop vs. simplicity — see [Design](#design)). + +### Non-goals + +- **HTTP/L7 ingress.** Public HTTP(S) is served by the existing tenant Ingress / Gateway API. +- **A platform-managed, always-on VPN service.** These are tenant-deployed catalog apps; a platform-run variant is possible future work. +- **A per-tenant egress IP.** Deferred future work; the gateway VM is its natural home later. +- **Routing a remote CIDR onto the shared default-VPC router** — rejected (cross-tenant collisions). Routed mode uses tenant-scoped kube-ovn routes, never the shared router. + +## Design + +### Two apps over one foundation + +Both apps share roughly 80% of their implementation — the VM appliance, the tunnel backend, the #29 exposure of the tunnel endpoint, the day-2 configuration mechanism, secret handling, and packaging. They differ only in what happens to a decrypted packet (L3-forward vs DNAT/SNAT) and in the CNI coupling that entails: + +| Axis | `site-gateway` (NAT) | `site-router` (routed) | +|------|----------------------|------------------------| +| Inner path | DNAT + SNAT → managed-app `ClusterIP` | plain L3 forwarding, no NAT | +| Return path | SNAT masquerade | kube-ovn `ovn.kubernetes.io/routes` annotation | +| CNI coupling | none — any CNI | kube-ovn-specific (routes + port_security) | +| Source IP | not preserved (SNAT) | preserved | +| Reachability | per host:port | whole subnet + ICMP | +| Config surface | `exposedServices` / `remoteTargets` (per-target ports) | `remoteCIDRs` / static routes / BGP | +| Security posture | fully contained | port_security relaxed (scoped is the target state) | +| Topology-advertising apps | need per-member advertised addresses | transparent | + +Why two apps rather than one `mode:` toggle: the two have divergent values schemas (per-target ports vs CIDRs/BGP), asymmetric controllers (routed needs a privileged CNI-mediation controller; NAT needs none), and different security postures a cluster admin may want to gate independently. Sharing the implementation (a common VyOS-driving controller core + image + base chart) keeps this from becoming duplicated code. + +### Shared foundation + +**Principle.** Terminate the tunnel in a KubeVirt VM (contained privilege). The VM is dual-homed on the tenant's default pod network and is the sole holder of the tunnel. + +```mermaid +flowchart TB + ext["External site
(on-prem router / cloud VPN / another org)"] + xp["#29 tunnel exposure
(ServiceExposure, UDP)"] + subgraph node["Worker node"] + gw["Gateway VM (VyOS)
tunnel termination + firewall + MSS clamp
+ (NAT: DNAT/SNAT) / (routed: L3 forward)"] + a["managed app A"] + b["managed app B"] + end + ext -->|"IPsec ESP-in-UDP / WireGuard UDP"| xp --> gw + gw --> a & b +``` + +**Tunnel backend — IPsec or WireGuard.** VyOS terminates both natively and `tunnel.type` selects per peer. **WireGuard is the simpler default on an overlay CNI** — UDP-native (a single configurable UDP port), so it rides the pod overlay as ordinary UDP with no ESP, no NAT-T, and no forced encapsulation; static keypairs, no IKE, smaller header overhead (see the validated finding in [Testing](#testing)). **IPsec is for interop** — much external/enterprise gear only speaks IKEv2/IPsec; supported, with the encapsulation requirement in [Testing](#testing). + +**Tunnel entry point.** Published via #29 `ServiceExposure` + `ExposureClass` (UDP), not a bespoke `LoadBalancer` — one shared external-address contract for both apps. + +**Day-2 configuration.** cloud-init provisions the VM only at first boot (KubeVirt bakes it into a static disk that is regenerated only on VM restart), so ongoing changes — add/rotate a peer, add a target or route — are driven by a **platform controller talking to the guest's own management API**. For VyOS the HTTPS API applies an atomic set/delete batch, so a change touches only the affected config and unrelated tunnels stay up. The management-API key is generated at runtime and seeded once via first-boot cloud-init; the API is reachable by the platform only (see [Security](#security)). A first iteration may fall back to "config change = VM restart" if called out explicitly. + +**Controller & low-level entities.** Tenant input stays in the catalog-app values (no new CRD); validation is the values schema plus controller-side reconcile checks surfaced as status conditions. A platform controller reconciles the VM + tunnel and pushes rendered config to the guest. The controller is split behind a small backend interface so the neutral VyOS-driving core (render → push → observe → status) is shared, and each app implements only its own materialization and any CNI mediation. The two apps differ sharply here: `site-router` carries a CNI-mediation controller (below); `site-gateway` needs none. + +**Image.** A VyOS appliance image (IPsec, WireGuard, native DNAT/SNAT, routing, and firewalling in one network OS), published by the platform as a reproducible, KubeVirt-consumable artifact. + +### `site-gateway` — NAT mode + +**Inbound (external site → any tenant app).** The remote peer targets a virtual service-exposure address that the gateway owns and maps to an app `ClusterIP`: + +1. The remote peer sends tunnel traffic to the gateway's #29-exposed UDP endpoint; the VM decrypts the tunnel. +2. **DNAT**: the virtual destination → the target app's `ClusterIP:port` (one entry per exposed app — the exposure table). +3. **SNAT (masquerade)**: source → the gateway's own pod IP. Mandatory: the cluster has no route back to the remote peer's inner IP, so without SNAT the app's reply leaves via the default route and is black-holed. +4. The VM forwards onto the pod network; the node resolves the `ClusterIP` to a backend pod; the app sees the gateway pod IP as client. Replies retrace the path and are re-encrypted. + +**Outbound (any tenant app → remote service over tunnel).** The app talks to a **local Service** whose endpoint is the gateway VM (the only app-side change is the target hostname). The node routes that `ClusterIP` to the gateway on a **per-target-unique listener port** (the local Service still exposes the standard port, mapped to a unique port on the VM), so the VM can disambiguate multiple remote targets that share a destination port; the VM DNATs that unique port to the real remote `IP:port` and SNATs into the tunnel's inner subnet. + +**Why it generalizes.** The app only ever sees a `ClusterIP` on its own network and speaks no VPN; DNAT is L4 (TCP/UDP, any port), so it works for databases, queues, object storage, gRPC, and raw UDP. `site-gateway` is CNI-agnostic and fully contained — SNAT keeps every emitted packet on the gateway's own pod IP, so no port-security relaxation is needed. + +**Topology-advertising apps.** Protocols where a client bootstraps to one endpoint and is then handed the members' own addresses to connect to directly — Kafka (`advertised.listeners`), MongoDB replica-set/sharded, Redis Cluster, Cassandra/Scylla, Elasticsearch with sniffing — are **not** transparent under plain NAT of the bootstrap endpoint: each member needs its own tunnel-reachable address that the app also advertises. Under `site-gateway` this means explicit per-member mapping (a distinct tunnel-side address:port per member, plus the app configured to advertise it); the managed-app operators already do this kind of thing for public exposure, but #29's `target` is per-named-listener, not per-broker/replica, so #29 does not directly cover per-member addressing today. The transparent path for this class is therefore **`site-router` (routed)** — the members' real addresses are directly reachable, so discovery works as long as members advertise a routable ClusterIP. Folding per-member NAT into a tunnel-backed `ExposureClass` is a possible future unification, not a solved design (see [Open questions](#open-questions)). Single-endpoint protocols (Postgres, MySQL, plain Redis, AMQP, ClickHouse client, HTTP/gRPC) are transparent under NAT and need none of this. + +### `site-router` — routed mode + +The decrypted packet is forwarded onto the tenant network **without NAT**; the return path is the kube-ovn `ovn.kubernetes.io/routes` namespace annotation (inherited onto pods at creation by the kube-ovn webhook), so the **client source IP is preserved** and whole-subnet reachability + ICMP work. This is the mode for cases where the remote side authorizes by source IP, or needs to reach a whole subnet, or where the app advertises its own topology (members' real addresses are directly reachable). + +**CNI-mediation controller.** A tenant cannot annotate its own namespace or touch `port_security`, so the platform controller mediates: validate declared remote CIDRs (disjoint from pod/service/join/node networks; cross-tenant overlap is fine — routes are namespace-scoped), set the namespace routes annotation, scope/relax `port_security` on the gateway VM port, and clean everything up on deletion. Because the annotation is applied at pod **creation**, existing tenant workloads gain reachability only after they are rolled; the controller reports the pods whose annotation lags the namespace ("pods pending route") rather than restarting anything. + +**Security posture.** Routed mode relaxes `port_security` on the gateway port. Containment comes from Cilium's sender-side egress enforcement — the endpoint identity is compiled into the per-endpoint eBPF program, so a spoofed source cannot widen the reachable set beyond the tenant tree + world. The target state is **scoped** `port_security` (the declared remote CIDRs added to the port's allowed addresses). Residual risks are documented in [Security](#security). + +**Transport note.** Routed mode carries transparent L3 for TCP/UDP/ICMP/SCTP only — non-standard IP protocols (ESP, GRE, VRRP) still do not cross the pod fabric (same root cause as the ESP finding in [Testing](#testing)). It is kube-ovn-specific and requires the remote CIDR to be disjoint from cluster networks. + +## User-facing changes + +Two catalog apps appear in the tenant dashboard, sharing the tunnel/peer/resources values and differing only in the bridging block. Sketches (cozyvalues-gen annotated): + +```yaml +# site-gateway (NAT) +## @param tunnel.type {string} Backend: "wireguard" (default; UDP-native) or "ipsec". +tunnel: { type: wireguard } +## @param peer.address {string} Public address of the remote peer / WireGuard endpoint. +peer: + address: "" + auth: { secretRef: "" } # IPsec IKE/PSK-or-cert; WireGuard peer public key + allowedIPs +## @param exposedServices {array} Inbound: {name, listenPort, targetService, port} +exposedServices: [] +## @param remoteTargets {array} Outbound: {name, localPort, remoteHost, remotePort} +remoteTargets: [] +## @param resources {object} VM sizing (cpu/memory). +resources: { cpu: "1", memory: "1Gi" } +``` + +```yaml +# site-router (routed) +## @param tunnel.type {string} Backend: "wireguard" (default) or "ipsec". +tunnel: { type: wireguard } +## @param peer.address {string} Public address of the remote peer / WireGuard endpoint. +peer: + address: "" + auth: { secretRef: "" } +## @param remoteCIDRs {array} Remote networks reachable over the tunnel (must be disjoint from cluster networks). +remoteCIDRs: [] +## @param staticRoutes {array} Optional extra static routes. +staticRoutes: [] +## @param bgp {object} Optional BGP peering with the remote side. +bgp: { enabled: false } +## @param resources {object} VM sizing (cpu/memory). +resources: { cpu: "1", memory: "1Gi" } +``` + +A cluster admin can enable/disable `site-router` independently — its kube-ovn coupling and `port_security` relaxation may not be wanted on every cluster. No existing app, CRD, or API changes. + +## Upgrade and rollback compatibility + +Purely additive and opt-in. No migration; existing clusters, manifests, and APIs are unaffected. The gateway VM image is a new artifact built and published by the platform. Rollback is deletion of the app instance (its `VMInstance`/`VMDisk`, the generated Services, and — for `site-router` — the namespace annotation / port_security scoping the controller cleans up), which removes the gateway entirely; nothing else in the tenant is touched. + +## Security + +- **Contained privilege (both apps).** The privileged dataplane lives in a VM guest kernel isolated by hardware virtualization; a compromise inside the gateway cannot manipulate the host node's networking or observe other tenants, and the platform never hands a tenant a privileged host-cluster pod. +- **`site-gateway` is fully contained.** SNAT keeps every packet the gateway emits on its own pod IP, so the CNI's port-security never sees a foreign source and needs no relaxation. +- **`site-router` relaxes port_security** on the gateway port (scoped is the target state). Containment is Cilium's sender-side egress enforcement; residual risks to document: intra-namespace identity spoofing (receiver-side identity resolved from source IP) and spoofed-source egress where SNAT is absent. +- **Managed apps untouched**; a firewall allow-list on the gateway makes exposure explicit; all state is tenant-scoped. +- **Secret handling.** A tenant does not create `Secret` objects directly; as with the existing managed apps (`apps/vpn`, `postgres`, `mariadb`) the app chart templates the `Secret` into the tenant namespace from the app values — autogenerating material when it is not supplied and using `lookup` to keep it stable across reconciles. The design minimizes what must be shipped: for **WireGuard** the tenant's private key is generated in-chart (never entered) and only its public key is surfaced for the remote side, while the peer's public key + endpoint are ordinary non-secret values — so no tenant secret is shipped at all. The one genuinely secret tenant input is the **IPsec PSK** (it must match the remote peer, so it cannot be autogenerated); it rides a values field into the chart-templated `Secret`, with the same plaintext-in-values-at-rest exposure the platform is already addressing for database passwords. The management-API key is generated at runtime, never entered. +- **Management API is platform-only.** The guest's config API is network-restricted so only the platform controller can reach it. + +## Failure and edge cases + +- **Missing SNAT → inbound reply black-holed (`site-gateway`).** Without the source-NAT rule the app replies to an unroutable tunnel address and the flow times out. SNAT is mandatory (validated). +- **MTU / double encapsulation.** The overlay already lowers MTU and the tunnel adds overhead; without MSS clamping (and/or a lowered tunnel MTU) large TCP packets black-hole. The gateway sets an **MSS clamp by default**, derived from the detected overlay MTU — clamping is the chosen approach rather than leaving MTU to the tenant. +- **Native ESP dropped pod-to-pod (IPsec).** The CNI drops non-TCP/UDP/ICMP/SCTP IP protocols between pods — native ESP (proto 50) included — so ESP-in-UDP (forced UDP encapsulation) is required unconditionally (validated). In policy-enforced tenant pods the dropper is Cilium's conntrack ("Unknown L4 protocol"); it is not geneve-specific. WireGuard, being UDP-native, is unaffected. +- **DNAT must target stable `ClusterIP`s**, never ephemeral pod IPs (`site-gateway`). +- **Source IP is lost inbound under NAT** (`site-gateway`) — use `site-router` when the remote side needs it. +- **VM image must be bootable and 512-native.** DRBD-backed (4K-sector) StorageClasses cannot boot a 512-native GPT image; the disk must sit on a 512-native StorageClass or use a KubeVirt `blockSize` override (e.g. `blockSize.custom.logical: 512` / `physical: 4096` on the disk spec). The image must ship a real bootloader (validated the hard way — see Testing). +- **A single gateway is a per-tenant SPOF** until HA is added (see [High availability](#high-availability)). + +## Testing + +The design was **prototyped and validated end-to-end** on a development Cozystack cluster (KubeVirt, Cilium + kube-ovn, LINSTOR), using two gateway VMs — one acting as the tenant gateway, one simulating the external site — with a real tenant-managed Postgres as the inbound target. The **NAT (`site-gateway`) inbound/outbound paths** were validated with the IPsec backend; the **routed (`site-router`) path and the WireGuard backend** are the next to validate (open questions). All items passed: + +| # | Item | Result | +|---|------|--------| +| 1 | IKEv2 SA establishes both sides | PASS | +| 2 | Inbound: external site → virtual VIP → DNAT/SNAT → managed Postgres (real server response) | PASS | +| 3 | Outbound: gateway client → tunnel → remote listener | PASS | +| 4 | SNAT-required negative test (removing SNAT black-holes the reply) | PASS | +| 5 | MTU / MSS clamp (tunnel MTU 1320, clamp 1280; multi-MB transfer, no stall) | PASS | + +**Key finding — native ESP is dropped by the CNI overlay.** IKE (UDP) negotiated and the SA came up, but the ESP data plane was silently dropped even pod-to-pod (receiver saw zero ESP, 100% loss). Forcing ESP-in-UDP encapsulation on both peers restored bidirectional traffic. On an overlay CNI the IPsec backend must therefore always force UDP encapsulation — and **WireGuard, being UDP-native, sidesteps this entirely**, which is a strong argument for it as the default backend. + +**Non-standard-protocol drop (informs HA).** Independently re-verified with a minimal repro: the CNI drops *all* non-TCP/UDP/ICMP/SCTP IP protocols (proto 112/50/47/4) pod-to-pod, both same-node and cross-node — so it is **not** the geneve tunnel. In policy-enforced tenant pods the dropper is **Cilium's conntrack** (`bpf_lxc.c`, "CT: Unknown L4 protocol"); without a policy it is the OVS datapath. This is the same root cause as the native-ESP drop, and it is why keepalived/VRRP cannot elect over the pod network (see [High availability](#high-availability)). Separately, a kube-ovn `Vip` + `ovn.kubernetes.io/aaps` annotation was validated to share a VIP across two gateway pods with `port_security` kept **on** (the VIP moves on owner change and a bogus source is still dropped — anti-spoofing scoped, not disabled). + +Planned automated coverage before implementation lands: helm-unittest for chart rendering across backends and both apps; an e2e that stands up the two-VM topology and asserts the inbound/outbound flows plus the SNAT-required and MSS behaviors. + +## High availability + +Deferred for the first iteration: a **single gateway VM with KubeVirt live-migration** for planned maintenance (conntrack, tunnel SA, and pod IP preserved — no tunnel drop; requires migratable/replicated storage, see the `blockSize` caveat). Automatic **unplanned-failure** HA is a follow-up; the preferred path is **Service-fronted active/passive** because it is CNI-agnostic (no floating L2 VIP, so no port-security interaction) at the cost of a small leader-election/lease agent and seconds-scale endpoint reconvergence. A kube-ovn shared-VIP via allowed-address-pairs was validated to move a VIP with `port_security` kept on, but keepalived-style election cannot run over the pod network (VRRP/proto-112 is dropped — see [Testing](#testing)), so it would need a side-channel or controller to drive the VIP claim. + +## Rollout + +- **Phase 1 — `site-router` (routed), IPsec backend**: the routed dataplane plus its CNI-mediation controller (CIDR validation, routes annotation, scoped `port_security`), with **MSS clamping** and **tunnel-state observability** (SA up/down, rekey, counters) surfaced from the start. Routed is the primary mode tenants ask for. (Note: the end-to-end prototype so far validated the NAT dataplane, so routed carries the first round of validation here.) +- **Phase 2 — `site-gateway` (NAT)**: the inbound/outbound DNAT/SNAT machinery (prototype-validated) with forced UDP encapsulation. +- **Phase 3 — WireGuard backend**: validate over the overlay and, if confirmed simpler/robust, make it the default `tunnel.type`. +- **Phase 4 — the rest**: HA (Service-fronted active/passive), scoped `port_security` for routed, per-tenant egress IP. + +Ordering leads with routed because it is the primary requested mode; the shared foundation is built once and both apps sit on it. + +## Open questions + +- **IPsec PSK at rest.** WireGuard ships no tenant secret (its key is autogenerated in-chart); the IPsec PSK is the one tenant-supplied secret — decide whether it stays a values field templated into a `Secret` (like DB passwords today — plaintext in values at rest) or gets a write-only / dashboard-mediated affordance. +- **Tunnel exposure via #29.** Confirm the `ExposureClass` UDP support and how the tunnel endpoint address interacts with the tenant's LB pool and quotas. +- **Topology-advertising apps under NAT.** Whether per-member advertised addressing for Kafka / Mongo / Redis-Cluster / Cassandra under `site-gateway` is worth a tunnel-backed `ExposureClass` (needs per-member granularity #29 does not have today) or is simply left to `site-router`. +- **HA mechanism.** Given VRRP does not cross the overlay (see Testing), standardize on Service/endpoint-based failover (needs a controller/lease) vs an AAP shared-VIP with side-channel election. +- **Scoped port_security for `site-router`.** kube-ovn allowed-address-pairs need CIDR support to add the declared remote CIDRs to the port (OVN itself accepts `MAC IP/mask`). +- **WireGuard backend.** Validate the handshake/SA over the overlay and confirm identical bridging before defaulting to it. +- **Relationship to ClusterMesh (#7).** Whether to present these apps and ClusterMesh as one coordinated "tenant connectivity" story, and where the boundary sits. + +## Alternatives considered + +- **Give tenants privileged / `NET_ADMIN` host pods.** Rejected — tenant escape; defeats namespace hardening. The whole design exists to avoid this. +- **Route the remote CIDR into the shared default-VPC router.** Rejected — the router is shared cluster-wide; injecting tenant RFC1918 CIDRs causes cross-tenant collisions and leakage. (`site-router` instead uses tenant-scoped kube-ovn namespace routes.) +- **Put managed apps on a custom kube-ovn VPC.** Not viable — VPCs are isolated from the default network and managed-app operators reconcile over the default network; only VMs are dual-homed. +- **Platform-run kube-ovn VPC-NAT-Gateway with the tunnel.** Possible future managed variant, but more platform work and it couples XFRM to the host kernel + OVS. The VM approach contains the privilege more cleanly and ships sooner. +- **A single app with a `mode: routed|nat` toggle.** Rejected — the two modes have divergent values schemas, asymmetric controllers (routed needs a privileged CNI-mediation controller, NAT needs none), and different security postures a cluster admin may want to gate independently. Two apps over a shared implementation express this more cleanly than conditional schema + always-on machinery. +- **Traefik / L7 proxy.** Limited — HTTP-only; not general across L4 protocols. + +### ClusterMesh / Kilo (#7), and why two mechanisms + +[#7](https://github.com/cozystack/community/pull/7) pursues an overlapping goal (tenant access to services across a boundary) but is a different shape, and neither mechanism subsumes the other: + +| Axis | ClusterMesh / Kilo (#7) | This proposal (gateway VMs) | +|------|-------------------------|-----------------------------| +| Remote end | A cooperative Kubernetes cluster running Kilo, reached via a kubeconfig | An arbitrary external site — not Kubernetes, no Kilo, no kubeconfig | +| Tenant workload served | The tenant's managed Kubernetes cluster (guest-VM pods) | Managed apps in the host-cluster tenant namespace | +| Transport | WireGuard node-to-node mesh (`mesh-granularity=cross`) | IPsec or WireGuard site-to-site, one VM | +| Address plane | Routed pod-CIDRs, no NAT; disjoint CIDRs required | `site-router`: routed (source-IP preserved); `site-gateway`: NAT to ClusterIPs (tolerates overlap) | +| Throughput | Full node×node mesh (built for Ceph) | Single gateway — app-level flows | +| Privilege | Kilo `NET_ADMIN` inside guest-cluster VMs → contained | `NET_ADMIN` inside the gateway VM guest → contained | + +Kilo cannot terminate a third-party VPN (both ends must run Kilo with a kubeconfig), and a single gateway VM cannot serve the Ceph-scale mesh. They are different layers — a platform storage fabric vs. a tenant-edge VPN concentrator — analogous to a cloud provider shipping both VPC Peering and a VPN Gateway. Both share the principle that matters: no privileged pods in the shared host namespace; privilege is contained in guest-cluster VMs (Kilo) or a dedicated gateway VM (this proposal). diff --git a/design-proposals/tenant-site-gateway/README.md b/design-proposals/tenant-site-gateway/README.md deleted file mode 100644 index 85878c5..0000000 --- a/design-proposals/tenant-site-gateway/README.md +++ /dev/null @@ -1,238 +0,0 @@ -# Tenant-managed external network connectivity via a gateway VM (IPsec / WireGuard) - -- **Title:** `Tenant-managed external network connectivity via a gateway VM (IPsec / WireGuard)` -- **Author(s):** `@myasnikovdaniil` -- **Date:** `2026-07-07` -- **Status:** Draft - -## Overview - -Cozystack tenants need site-to-site VPN connectivity between their workloads and external networks, in both directions: exposing a tenant-managed application to a remote site (inbound), and letting a tenant-managed application reach a service that lives only behind a tunnel (outbound). This must work for any application — databases, message queues, object storage, gRPC, raw UDP — not one special case. - -This proposal terminates the tunnel (IPsec or WireGuard) **inside a KubeVirt VM** rather than in a pod. The privileged dataplane (`NET_ADMIN`, XFRM / a `wireguard` interface, `ip_forward`) lives in a guest kernel isolated by hardware virtualization, so its blast radius is contained by the VM boundary — the host node and other tenants are unaffected. The gateway VM is a normal member of the tenant's pod network, so it reaches any tenant `ClusterIP` Service; it performs all address translation (DNAT/SNAT), and the managed applications are never modified and never speak the VPN — they only ever see ordinary `ClusterIP`s on their own network. We propose shipping this as a Cozystack catalog app, provisionally `packages/apps/site-gateway`. - -The design has been prototyped and validated end-to-end (see [Testing](#testing)). - -## Scope and related proposals - -- **Related — [`cross-cluster-tenant-mesh` (ClusterMesh / Kilo)](https://github.com/cozystack/community/pull/7).** That proposal connects cooperative Kubernetes clusters into a routed WireGuard node-to-node mesh for high-throughput cases (its motivating example is a tenant cluster consuming host Ceph). This proposal is complementary and occupies a different niche — an arbitrary external site (not a Kubernetes cluster), NAT-bridged to managed apps that live in the tenant namespace. See [Alternatives considered](#alternatives-considered) for a side-by-side and a "why two mechanisms" discussion. Notably, PR #7 explicitly defers the "tenant ↔ tenant-namespace applications" and "NAT-egress" integration — precisely the space this proposal fills. -- **Deferred here:** a platform-run (non-VM) managed variant; HTTP/L7 ingress (the existing tenant Ingress / Gateway API already covers public HTTP); preservation of the original client source IP inbound. - -## Context - -Cozystack runs tenant workloads under a hardened networking model: - -- **Namespace hardening.** Cozystack applies Pod Security Standards to tenant namespaces. The platform operator only sets `pod-security.kubernetes.io/enforce=privileged` on a namespace when a platform-declared package requests it. A regular tenant cannot schedule a privileged, `NET_ADMIN`, or `hostNetwork` pod — deliberately, because such a pod executes against the node kernel and could manipulate host networking or observe other tenants' traffic. -- **Default pod network vs. custom VPC.** Managed apps (databases and friends) run as ordinary pods on the **default cluster pod network** and are reached via `ClusterIP` Services + CoreDNS. They are not placed on a custom kube-ovn VPC; VPCs are isolated from the default network and the managed-app operators reconcile over the default network — so a managed app cannot simply be "moved onto a VPC." -- **Dual-homed VMs.** KubeVirt VMs are the exception: every VM always has a `default` pod-network NIC (and can carry additional multus interfaces). A VM is therefore a first-class member of the default pod network. This is the primitive the design leans on. - -### The problem - -A tenant today has no supported way to: - -- **Inbound:** let a remote site reach a tenant-managed app (a Postgres instance, an internal API, an object store) over a tunnel, without exposing it to the public internet. -- **Outbound:** let a tenant app call a service (a legacy database, a licensing server, an on-prem API) that is only routable through a tunnel. - -The naive approaches all require either a privileged host-cluster pod (a tenant-escape vector — defeats the hardening above) or changes to shared, cluster-wide network infrastructure (cross-tenant collision risk). We want a self-service, tenant-scoped mechanism that generalizes to any protocol and touches neither the host kernel nor the managed apps. - -## Goals - -- Give tenants **self-service** site-to-site connectivity, both directions, through a catalog app. -- Support **inbound** exposure of any tenant-managed app to a remote site over the tunnel. -- Support **outbound** connections from any tenant app to services reachable only via the tunnel. -- Be **app-agnostic** — work for any L4 protocol (TCP/UDP, any port) without modifying the target apps. -- Keep the privileged dataplane **contained** in a VM guest; add no privileges to the host cluster and none to managed apps. -- Keep all bridging/NAT **tenant-scoped**; make no changes to shared cluster routers. -- Support **both IPsec and WireGuard** as tunnel backends (interop vs. simplicity — see [Design](#design)). - -### Non-goals - -- **HTTP/L7 ingress.** Public HTTP(S) is served by the existing tenant Ingress / Gateway API; the gateway VM is for site-to-site VPN and non-HTTP protocols. -- **Preserving the original client source IP inbound.** SNAT hides it (see [Failure and edge cases](#failure-and-edge-cases)); source-IP ACLs are out of scope for the first iteration. -- **A platform-managed, always-on VPN service.** This is a tenant-deployed catalog app; a platform-run variant is possible future work. -- **Routing a remote CIDR onto the shared default-VPC router** — explicitly rejected (cross-tenant collisions). - -## Design - -### Principle - -Terminate the tunnel in a KubeVirt VM. The VM guest kernel isolates `NET_ADMIN`/XFRM from the host, so the tenant effectively runs a privileged network appliance without the platform ever granting a privileged pod. The VM is dual-homed on the tenant's default pod network and is the sole holder of the tunnel; it DNAT/SNATs between the tunnel and the tenants' managed-app `ClusterIP`s. - -### Topology - -```mermaid -flowchart TB - ext["External site
(on-prem router / cloud VPN / another org)"] - lb["LoadBalancer VIP
(tunnel UDP ports)"] - subgraph node["Worker node"] - gw["Gateway VM (VyOS)
tunnel termination + DNAT/SNAT + firewall + MSS clamp"] - a["managed app A
(ClusterIP)"] - b["managed app B
(ClusterIP)"] - c["managed app C
(ClusterIP)"] - end - ext -->|"IPsec ESP-in-UDP / WireGuard UDP"| lb --> gw - gw --> a & b & c -``` - -Everything privileged lives inside the VM guest. Managed apps only ever see `ClusterIP`s on their own default network. - -### Inbound path (external site → any tenant app) - -The remote peer targets a virtual "service-exposure" address that the gateway owns and maps to an app `ClusterIP`. - -1. Remote peer sends tunnel traffic to the gateway's public VIP (a `LoadBalancer` Service on the tunnel's UDP ports). -2. The VIP delivers to the gateway VM's pod NIC; the VM decrypts the tunnel. -3. **DNAT**: the virtual destination (e.g. `10.200.0.10:5432`) → the target app's `ClusterIP:port`. One entry per exposed app — the exposure table. -4. **SNAT (masquerade)**: source → the gateway's own pod IP. Mandatory: the cluster has no route back to the remote peer's inner IP, so without SNAT the app's reply would leave via the default route and be black-holed. -5. The VM forwards onto the pod network; the node resolves the `ClusterIP` to a backend pod; the app sees the connection with the gateway pod IP as client. Replies retrace the path and are re-encrypted back over the tunnel. - -### Outbound path (any tenant app → remote service over tunnel) - -The tenant app never routes to the remote address directly; it talks to a **local Service** whose endpoint is the gateway VM. - -1. The app connects to a local Service name the chart creates (e.g. `remote-db..svc:port`) — the only app-side change is the target hostname. -2. The node routes the `ClusterIP` to the gateway VM on a **per-target-unique listener port** (the local Service still exposes the standard port, e.g. 5432, mapped to a unique port on the VM). The uniqueness is required so the VM can tell apart multiple remote targets that share the same destination port. -3. **DNAT**: that unique local listener port → the real remote `IP:port` behind the tunnel; the VM encrypts and sends it over the tunnel; SNAT into the tunnel's inner subnet lets replies return. - -**Rejected alternative — a static route on the shared default-VPC router.** Injecting a tenant's remote RFC1918 CIDR into the cluster-wide default router risks cross-tenant address collisions and leakage. The local-Service approach keeps every remote address strictly inside the tenant's gateway VM. - -### Why it generalizes to all apps - -- The app only ever sees a `ClusterIP` / Service on its own network; it speaks no VPN and needs no route to the far side. -- DNAT is **L4** (TCP and UDP, any port), not L7 — so it works for databases, queues, object storage, gRPC, and raw UDP, unlike an HTTP-only path. -- The gateway VM is the only privileged component, and it is a contained VM guest. - -### Tunnel backend: IPsec or WireGuard - -VyOS terminates both natively and the DNAT/SNAT bridging is identical — only the transport differs. `tunnel.type` selects the backend per peer. - -- **WireGuard — the simpler default on an overlay CNI.** WireGuard is UDP-native (a single configurable UDP port), so it rides the pod overlay as ordinary UDP: no ESP, no NAT-T, no forced encapsulation. It is simpler (static keypairs, no IKE) and has smaller header overhead. Preferred for tenant↔tenant and tenant↔Cozystack links. (See the validated finding in [Testing](#testing) that makes this concrete.) -- **IPsec — for interop.** Much external/enterprise gear only speaks IKEv2/IPsec; when the remote site is not under the tenant's control this is often the only option. Supported, with the encapsulation requirement described in [Testing](#testing). - -### Build options - -- **VyOS appliance (recommended).** A single network OS does IPsec, WireGuard, native DNAT/SNAT, and firewalling; no separate proxy needed; configured declaratively via cloud-init. -- **Alpine + libreswan/strongSwan (or wireguard-tools) + haproxy.** A leaner, self-composed image if a network-OS dependency is undesirable; more moving parts. - -### High availability - -HA splits by failure mode: - -- **Planned maintenance (node drain):** KubeVirt **live-migration** relocates the gateway VM with its state intact (conntrack, tunnel SA, pod IP preserved) — no tunnel drop. Preferred for maintenance; needs no standby. (Requires migratable storage — see the storage caveat below.) -- **Unplanned node/VM failure:** needs a standby plus a failover trigger. Two mechanisms, both examined on kube-ovn (see [Testing](#testing) for the shared-VIP validation): - - 1. **Service-fronted active/passive.** The external `LoadBalancer` (tunnel) and internal `ClusterIP` (outbound) Services select whichever gateway pod is active; failover = re-pointing endpoints when the active goes unready. The stable address is the Service VIP (CNI-managed) — no floating L2 VIP, so no port-security interaction at all. Cost: needs a small leader-election/lease agent (or controller) to flip the active endpoint, and failover takes seconds (endpoint reconvergence). CNI-agnostic. - 2. **Shared VIP via kube-ovn allowed-address-pairs (AAP).** A kube-ovn `Vip` plus the `ovn.kubernetes.io/aaps` pod annotation adds *only* the VIP to the port's OVN `port_security` (which stays enforced) — validated: the VIP is reachable, moves between the two gateway pods through OVN on owner change, and any other source address is still dropped, so anti-spoofing is **scoped rather than disabled** and the VM stays fully tenant-controlled (no need to withhold direct VM/cloud-init access). **Caveat:** VRRP advertisements (IP proto 112) do not reach the peer pod, so keepalived split-brains and cannot elect a master over the pod network. Root cause (independently verified): the CNI drops *all* non-TCP/UDP/ICMP/SCTP IP protocols pod-to-pod — in policy-enforced tenant pods this is **Cilium's conntrack** (`bpf_lxc.c`, "CT: Unknown L4 protocol"), and without a policy it is the OVS datapath; it is **not** geneve/inter-node-specific (same-node pods drop it too). This is the same root cause that makes native ESP need UDP encapsulation. To use this path the election must therefore run over a side-channel (e.g. the gateway pair's own tunnel link) or be driven by a controller; the AAP VIP move itself is proven. Also the `aaps` annotation must be baked into the KubeVirt VM's pod template (the current `VMInstance` chart does not expose it), and the VIP rides the pod's real MAC (keep the VyOS default — no virtual-MAC). - -**Recommendation:** iteration 1 ships a single gateway VM with live-migration for maintenance, and treats automatic unplanned-failure HA as a follow-up — neither mechanism is a clean drop-in today (Service-based needs a controller; AAP needs the VRRP-advert-delivery issue resolved). - -**Storage caveat:** live-migration and single-VM reschedule need migratable/replicated storage — but replicated (DRBD) StorageClasses expose 4K sectors, so the image needs the `blockSize` override to boot (see [Failure and edge cases](#failure-and-edge-cases)). A two-VM active/passive pair can instead use node-local 512-native disks. - -## User-facing changes - -A new catalog app, provisionally **`packages/apps/site-gateway`** (VM-backed), appears in the tenant dashboard. The tenant fills a values form; the platform-authored chart renders the gateway VM, its tunnel + NAT + firewall config, the local Services for outbound targets, and a `LoadBalancer` Service for the tunnel's UDP ports. Sketch of the values schema (cozyvalues-gen annotated): - -```yaml -## @param tunnel.type {string} Backend: "wireguard" (default; UDP-native) or "ipsec". -tunnel: - type: wireguard - -## @param peer.address {string} Public address of the remote peer / WireGuard endpoint. -peer: - address: "" - ## backend-specific auth: IPsec IKE/PSK-or-cert; WireGuard peer public key + allowedIPs. - auth: - secretRef: "" - -## @param exposedServices {array} Inbound: {name, listenPort, targetService, port} -exposedServices: [] - -## @param remoteTargets {array} Outbound: {name, localPort, remoteHost, remotePort} -remoteTargets: [] - -## @param resources {object} VM sizing (cpu/memory). -resources: - cpu: "1" - memory: "1Gi" -``` - -No existing app, CRD, or API changes. - -## Upgrade and rollback compatibility - -Purely additive and opt-in. No migration; existing clusters, manifests, and APIs are unaffected. The gateway VM image is a new artifact built and published by the platform. Rollback is deletion of the app instance (its `VMInstance`/`VMDisk` and the generated Services), which removes the gateway entirely; nothing else in the tenant is touched. - -## Security - -- **Contained privilege.** The privileged dataplane lives in a VM guest kernel isolated by hardware virtualization; a compromise or misconfiguration inside the gateway cannot manipulate the host node's networking or observe other tenants. The platform never hands a tenant a privileged host-cluster pod. -- **Managed apps untouched.** No new privileges or config on the apps; they stay ordinary pods on the default network. -- **Tenant-scoped bridging.** All translation is NAT inside the VM; nothing changes on the shared cluster router, so there is no cross-tenant surface. -- **Firewall allow-list.** The gateway restricts which tunnel-side sources may reach which exposed ports — exposure is explicit, not "the whole tunnel reaches everything." -- **Tenant-supplied secrets.** PSK / certificate / WireGuard key material is provided via a Secret reference and mounted into the guest; the chart must avoid persisting it in plaintext cloud-init user-data at rest (open question). - -## Failure and edge cases - -- **Missing SNAT → inbound reply black-holed.** Without the source-NAT rule the app replies to an unroutable tunnel address and the flow times out. SNAT is mandatory (validated). It also keeps the gateway **anti-spoofing-clean**: every packet it emits onto the pod network carries its own pod IP, so the CNI's port-security never sees a foreign source — the dataplane needs no port-security relaxation (that only arises for shared-VIP HA; see High availability). -- **MTU / double encapsulation.** The overlay already lowers MTU and the tunnel adds overhead; without MSS clamping (and/or a lowered tunnel MTU) large TCP packets black-hole. The gateway sets an MSS clamp by default. -- **Native ESP dropped pod-to-pod (IPsec).** The CNI drops non-TCP/UDP/ICMP/SCTP IP protocols between pods — native ESP (proto 50) included — so ESP-in-UDP (forced UDP encapsulation) is required unconditionally (validated). In policy-enforced tenant pods the dropper is Cilium's conntrack ("Unknown L4 protocol"); it is not geneve-specific (same root cause as the VRRP/proto-112 case — see Testing). WireGuard, being UDP-native, is unaffected. -- **DNAT must target stable `ClusterIP`s**, never ephemeral pod IPs. -- **Source IP is lost inbound** (SNAT) — only relevant for apps with source-IP ACLs. -- **VM image must be bootable and 512-native.** DRBD-backed (4K-sector) StorageClasses cannot boot a 512-native GPT image; the disk must sit on a 512-native StorageClass or use a KubeVirt `blockSize` override (e.g. `blockSize.custom.logical: 512` / `physical: 4096` on the disk spec). The image must ship a real bootloader (validated the hard way — see Testing). -- **Single gateway is a per-tenant SPOF** until HA is added (open question). - -## Testing - -The design was **prototyped and validated end-to-end** on a development Cozystack cluster (KubeVirt, Cilium + kube-ovn, LINSTOR), using two gateway VMs — one acting as the tenant gateway, one simulating the external site — with a real tenant-managed Postgres as the inbound target. IPsec was validated first; WireGuard is the next backend to validate (open question). All items passed: - -| # | Item | Result | -|---|------|--------| -| 1 | IKEv2 SA establishes both sides | PASS | -| 2 | Inbound: external site → virtual VIP → DNAT/SNAT → managed Postgres (real server response) | PASS | -| 3 | Outbound: gateway client → tunnel → remote listener | PASS | -| 4 | SNAT-required negative test (removing SNAT black-holes the reply) | PASS | -| 5 | MTU / MSS clamp (working tunnel MTU 1320, clamp 1280; multi-MB transfer, no stall) | PASS | - -**Key finding — native ESP is dropped by the CNI overlay.** IKE (UDP) negotiated and the SA came up, but the ESP data plane was silently dropped even pod-to-pod (receiver saw zero ESP, 100% loss). Forcing ESP-in-UDP encapsulation on both peers restored bidirectional traffic. Consequence: on an overlay CNI the IPsec backend must always force UDP encapsulation — and **WireGuard, being UDP-native, sidesteps this entirely**, which is a strong argument for it as the default backend. - -**HA mechanism validation (kube-ovn AAP).** Separately validated (kube-ovn v1.15.10): a `Vip` + `ovn.kubernetes.io/aaps` pod annotation shares a VIP across the two gateway VM pods with `port_security` kept **on**. Proven: (1) without AAP the VIP is dropped (GARP/ping blocked); (2) with AAP the VIP is reachable and lands on the active pod; (3) forcing the active down moves the VIP to the standby through OVN; (4) a bogus source address from the VM is still dropped — anti-spoofing is scoped, not disabled. **Not** working: VRRP advertisements (IP proto 112) do not reach the peer pod, so keepalived cannot elect over the pod network — the election needs a side-channel or controller (see [High availability](#high-availability)). Independently re-verified with a minimal repro: the CNI drops *all* non-TCP/UDP/ICMP/SCTP IP protocols (112, 50, 47, 4) pod-to-pod, both same-node and cross-node — so this is **not** the geneve tunnel. In policy-enforced tenant pods the dropper is **Cilium's conntrack** (`bpf_lxc.c`, "CT: Unknown L4 protocol"); without a policy it is the OVS datapath. This is the same root cause as the native-ESP drop above. An upstream issue, if pursued, targets Cilium (a known conntrack limitation); the OVS-side drop is not yet root-caused. - -Planned automated coverage before implementation lands: helm-unittest for chart rendering across backends; an e2e that stands up the two-VM topology and asserts the inbound/outbound flows and the SNAT-required and MSS behaviors. - -## Rollout - -- **Phase 1 — IPsec backend** (prototype validated): ship `site-gateway` with the IPsec backend (forced UDP encapsulation) + the inbound/outbound NAT machinery. -- **Phase 2 — WireGuard backend**: validate WireGuard over the overlay and, if confirmed simpler/robust, make it the default `tunnel.type`. -- **Phase 3 — hardening**: HA (see [High availability](#high-availability) — Service-based or AAP shared-VIP), tunnel-state observability in the dashboard, MSS/MTU auto-tuning. - -## Open questions - -- **Secret handling.** Deliver PSK/cert/WireGuard-key material to the guest without persisting it in plaintext cloud-init at rest — candidate: mount the referenced `Secret` as a separate config disk (or fetch it on first boot via a small init) instead of inlining it in the VM spec; the post-reconciliation read-access boundary needs to be defined. -- **VIP allocation.** One LoadBalancer VIP per gateway vs. sharing; interaction with the tenant's LB address pool and quotas. -- **HA mechanism.** Given VRRP advertisements do not cross the kube-ovn overlay (see Testing), which failover trigger to standardize on — Service/endpoint-based (needs a controller/lease) or AAP shared-VIP with election over a side-channel — and baking the `aaps` annotation into the `VMInstance` chart's pod template. -- **Observability.** Surfacing tunnel state (SA up/down, rekey, counters) through existing dashboards. -- **MTU auto-tuning.** Derive the MSS clamp / tunnel MTU from the detected overlay MTU, or require an explicit value. -- **WireGuard backend.** Validate the handshake/SA over the overlay and confirm identical DNAT/SNAT bridging before defaulting to it. -- **Relationship to ClusterMesh (PR #7).** Whether to present this and ClusterMesh as one coordinated "tenant connectivity" story, and where the boundary sits. - -## Alternatives considered - -- **Give tenants privileged / `NET_ADMIN` host pods.** Rejected — tenant escape; defeats namespace hardening. The whole design exists to avoid this. -- **Route the remote CIDR into the shared default-VPC router.** Rejected — the router is shared cluster-wide; injecting tenant RFC1918 CIDRs causes cross-tenant collisions and leakage. -- **Put managed apps on a custom kube-ovn VPC.** Not viable — VPCs are isolated from the default network and managed-app operators reconcile over the default network; only VMs are dual-homed. -- **Platform-run kube-ovn VPC-NAT-Gateway with the tunnel.** Possible future managed variant, but more platform work and it couples XFRM to the host kernel + OVS. The VM approach contains the privilege more cleanly and ships sooner. -- **Traefik / L7 proxy.** Limited — HTTP-only; not general across L4 protocols. - -### ClusterMesh / Kilo (PR #7), and why two mechanisms - -[PR #7](https://github.com/cozystack/community/pull/7) pursues an overlapping goal (tenant access to services across a boundary) but is a different shape, and neither mechanism subsumes the other: - -| Axis | ClusterMesh / Kilo (PR #7) | This proposal (gateway VM) | -|------|---------------------------|----------------------------| -| Remote end | A cooperative Kubernetes cluster running Kilo, reached via a kubeconfig | An arbitrary external site — not Kubernetes, no Kilo, no kubeconfig | -| Tenant workload served | The tenant's managed Kubernetes cluster (guest-VM pods) | Managed apps in the host-cluster tenant namespace (ClusterIPs) | -| Transport | WireGuard node-to-node mesh (`mesh-granularity=cross`) | IPsec or WireGuard site-to-site, one VM | -| Address plane | Routed pod-CIDRs, no NAT; disjoint CIDRs required | NAT (DNAT/SNAT) to ClusterIPs; tolerates overlap | -| Throughput | Full node×node mesh — direct pod-IP to many backends (built for Ceph) | Single gateway — app-level flows, not Ceph-scale | -| Privilege | Kilo node `NET_ADMIN`, on the tenant side inside guest-cluster VMs → contained | `NET_ADMIN` inside the gateway VM guest → contained | - -**Kilo cannot terminate a third-party VPN** (both ends must run Kilo with a kubeconfig), and **a single gateway VM cannot serve the Ceph-scale mesh** (throughput/topology incompatible with funneling). They are different layers: a platform storage fabric vs. a tenant-edge VPN concentrator — analogous to a cloud provider shipping both VPC Peering and a VPN Gateway. Both share the principle that matters here: no privileged pods in the shared host namespace; privilege is contained in guest-cluster VMs (Kilo) or a dedicated gateway VM (this proposal). Recommended framing: lead with the gateway VM as the single tenant-facing primitive (covers external sites, tenant↔tenant, and managed-app exposure), and use ClusterMesh only for the high-throughput cross-cluster storage case. From 48ebb1a79b568c52e21d1e4ec27383752e5da74b Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Wed, 15 Jul 2026 15:37:03 +0500 Subject: [PATCH 4/5] design-proposal: address review feedback on tenant site connectivity - Status Draft -> Review - Declare #29 (with UDP ExposureClass) an explicit Phase 1 blocker, with a stopgap UDP LoadBalancer fallback if it slips - Reframe routed (site-router) as validated in a production implementation; make scoped port_security a Phase 1 acceptance criterion, not a Phase 4 relaxation - Add an Image lifecycle subsection: VyOS image built in-repo (Talos-image pattern), CVE patching via re-pin+rebuild, update = new disk => VM restart - Specify tenant-immutable enforcement for the guest management API (no tenant-namespace NetworkPolicy; API key in a platform-only Secret), to be re-verified against the production implementation - Commit to singular peer: (one tunnel per instance); add a multi-peer open question instead of retrofitting peers:[] later - Enrich the remote-CIDR deny-set (link-local/metadata, LB pool, loopback, default route) and surface rejection as a status condition - Keep IPsec the default backend; WireGuard is a first-class alternative in Phase 3, never promoted to default Co-Authored-By: Claude Opus 4.8 (1M context) Signed-off-by: Myasnikov Daniil --- .../tenant-site-connectivity/README.md | 64 ++++++++++++------- 1 file changed, 40 insertions(+), 24 deletions(-) diff --git a/design-proposals/tenant-site-connectivity/README.md b/design-proposals/tenant-site-connectivity/README.md index 9989739..4342553 100644 --- a/design-proposals/tenant-site-connectivity/README.md +++ b/design-proposals/tenant-site-connectivity/README.md @@ -3,7 +3,7 @@ - **Title:** `Tenant-managed site-to-site connectivity via gateway VMs (site-router and site-gateway)` - **Author(s):** `@myasnikovdaniil` - **Date:** `2026-07-08` -- **Status:** Draft +- **Status:** Review ## Overview @@ -20,7 +20,7 @@ Neither is a fallback for the other — a cluster admin can offer either or both ## Scope and related proposals -- **[#29 structured external exposure](https://github.com/cozystack/community/pull/29).** The tunnel's own external entry point (the UDP listener the remote peer dials) is published through the structured `ServiceExposure` / `ExposureClass` primitives from #29 rather than a bespoke `LoadBalancer` path; that exposure class must support **UDP** (IKE/NAT-T 4500, WireGuard). +- **[#29 structured external exposure](https://github.com/cozystack/community/pull/29) — blocking dependency.** The tunnel's own external entry point (the UDP listener the remote peer dials) is published through the structured `ServiceExposure` / `ExposureClass` primitives from #29 rather than a bespoke `LoadBalancer` path, and that exposure class **must support UDP** (IKE/NAT-T 4500, WireGuard) — which #29 does not yet provide. This proposal declares #29 (with UDP support) an **explicit blocker for Phase 1**: implementation cannot land until #29 ships. If #29 stalls, the fallback is a bespoke UDP `LoadBalancer` Service on the gateway as a stopgap — explicitly a temporary path to be migrated onto the #29 contract once available, not a parallel long-term mechanism. - **[#7 ClusterMesh / Kilo](https://github.com/cozystack/community/pull/7).** Complementary and a different niche — a routed WireGuard node-to-node mesh between cooperative Kubernetes clusters. See [Alternatives considered](#alternatives-considered). - **Deferred here:** a platform-run (non-VM) managed variant; HTTP/L7 ingress (the existing tenant Ingress / Gateway API covers public HTTP); a per-tenant egress IP (future work — the gateway VM is its natural home later). @@ -65,7 +65,7 @@ Both apps share roughly 80% of their implementation — the VM appliance, the tu | Source IP | not preserved (SNAT) | preserved | | Reachability | per host:port | whole subnet + ICMP | | Config surface | `exposedServices` / `remoteTargets` (per-target ports) | `remoteCIDRs` / static routes / BGP | -| Security posture | fully contained | port_security relaxed (scoped is the target state) | +| Security posture | fully contained | port_security scoped to declared remote CIDRs (Phase 1 acceptance) | | Topology-advertising apps | need per-member advertised addresses | transparent | Why two apps rather than one `mode:` toggle: the two have divergent values schemas (per-target ports vs CIDRs/BGP), asymmetric controllers (routed needs a privileged CNI-mediation controller; NAT needs none), and different security postures a cluster admin may want to gate independently. Sharing the implementation (a common VyOS-driving controller core + image + base chart) keeps this from becoming duplicated code. @@ -87,15 +87,23 @@ flowchart TB gw --> a & b ``` -**Tunnel backend — IPsec or WireGuard.** VyOS terminates both natively and `tunnel.type` selects per peer. **WireGuard is the simpler default on an overlay CNI** — UDP-native (a single configurable UDP port), so it rides the pod overlay as ordinary UDP with no ESP, no NAT-T, and no forced encapsulation; static keypairs, no IKE, smaller header overhead (see the validated finding in [Testing](#testing)). **IPsec is for interop** — much external/enterprise gear only speaks IKEv2/IPsec; supported, with the encapsulation requirement in [Testing](#testing). +**Tunnel backend — IPsec or WireGuard.** VyOS terminates both natively and `tunnel.type` selects the backend. **IPsec is the default** — the remote side of a site-to-site tunnel is dictated by whatever the external site already runs, and much external/enterprise gear only speaks IKEv2/IPsec, so IPsec is the broadest-interop starting point (with the forced-UDP-encapsulation requirement in [Testing](#testing)). **WireGuard is the simpler backend on an overlay CNI** — UDP-native (a single configurable UDP port), so it rides the pod overlay as ordinary UDP with no ESP, no NAT-T, and no forced encapsulation; static keypairs, no IKE, smaller header overhead (see the validated finding in [Testing](#testing)). The default is not a strong statement of preference: the tenant picks the backend their site requires, so IPsec stays the default for interop breadth and WireGuard is a first-class alternative rather than a successor. **Tunnel entry point.** Published via #29 `ServiceExposure` + `ExposureClass` (UDP), not a bespoke `LoadBalancer` — one shared external-address contract for both apps. -**Day-2 configuration.** cloud-init provisions the VM only at first boot (KubeVirt bakes it into a static disk that is regenerated only on VM restart), so ongoing changes — add/rotate a peer, add a target or route — are driven by a **platform controller talking to the guest's own management API**. For VyOS the HTTPS API applies an atomic set/delete batch, so a change touches only the affected config and unrelated tunnels stay up. The management-API key is generated at runtime and seeded once via first-boot cloud-init; the API is reachable by the platform only (see [Security](#security)). A first iteration may fall back to "config change = VM restart" if called out explicitly. +**Day-2 configuration.** cloud-init provisions the VM only at first boot (KubeVirt bakes it into a static disk that is regenerated only on VM restart), so ongoing changes — rotate the peer's keys/PSK, add a target or route — are driven by a **platform controller talking to the guest's own management API**. For VyOS the HTTPS API applies an atomic set/delete batch, so a change touches only the affected config and the running tunnel stays up (each app instance terminates a single peer — see [Values schema](#user-facing-changes) and the multi-peer [open question](#open-questions)). The management-API key is generated at runtime and seeded once via first-boot cloud-init; the API is reachable by the platform only (see [Security](#security)). A first iteration may fall back to "config change = VM restart" if called out explicitly. **Controller & low-level entities.** Tenant input stays in the catalog-app values (no new CRD); validation is the values schema plus controller-side reconcile checks surfaced as status conditions. A platform controller reconciles the VM + tunnel and pushes rendered config to the guest. The controller is split behind a small backend interface so the neutral VyOS-driving core (render → push → observe → status) is shared, and each app implements only its own materialization and any CNI mediation. The two apps differ sharply here: `site-router` carries a CNI-mediation controller (below); `site-gateway` needs none. -**Image.** A VyOS appliance image (IPsec, WireGuard, native DNAT/SNAT, routing, and firewalling in one network OS), published by the platform as a reproducible, KubeVirt-consumable artifact. +**Image.** A VyOS appliance image (IPsec, WireGuard, native DNAT/SNAT, routing, and firewalling in one network OS), published by the platform as a reproducible, KubeVirt-consumable artifact. See [Image lifecycle](#image-lifecycle) for how it is built and updated. + +### Image lifecycle + +VyOS has no free LTS channel — only the rolling release is freely buildable — so the appliance image cannot be pinned to a vendor-maintained LTS and must be built and patched by the platform. This is handled the same way Cozystack already builds its other bootable-node artifacts: + +- **Build in the monorepo.** The image is built from a pinned VyOS rolling snapshot in-repo, alongside the existing node-image builds (the Talos image build is the live example; the previously shipped Ubuntu image build, now retired, followed the same shape). The build is reproducible and pinned to a specific upstream commit/date rather than tracking `latest`, so a given Cozystack release maps to an exact image. +- **Patch cadence.** Because the base is rolling, CVE and dependency patching happens by re-pinning to a newer VyOS snapshot and rebuilding as part of the normal monorepo release cadence — the same place Talos bumps land. Security-relevant rebuilds can be cut out-of-band on the same pipeline. +- **How a running gateway consumes an update.** The image backs a KubeVirt VM disk, so adopting a new image means booting the VM from the new disk — i.e. a **VM restart, which drops the tunnel** for the reconnect window. This is deliberately decoupled from **day-2 config** changes (peer keys, targets, routes), which are applied live via the guest management API and never require a restart (see the Day-2 paragraph above). Image updates are therefore the *only* routine tunnel-dropping operation; for planned image upgrades KubeVirt live-migration does **not** help (a new disk is a new boot, not a migration), so an image bump is an explicit maintenance action. Making it non-disruptive depends on the active/passive HA path (roll one gateway at a time behind the fronting Service) — see [High availability](#high-availability). ### `site-gateway` — NAT mode @@ -116,9 +124,9 @@ flowchart TB The decrypted packet is forwarded onto the tenant network **without NAT**; the return path is the kube-ovn `ovn.kubernetes.io/routes` namespace annotation (inherited onto pods at creation by the kube-ovn webhook), so the **client source IP is preserved** and whole-subnet reachability + ICMP work. This is the mode for cases where the remote side authorizes by source IP, or needs to reach a whole subnet, or where the app advertises its own topology (members' real addresses are directly reachable). -**CNI-mediation controller.** A tenant cannot annotate its own namespace or touch `port_security`, so the platform controller mediates: validate declared remote CIDRs (disjoint from pod/service/join/node networks; cross-tenant overlap is fine — routes are namespace-scoped), set the namespace routes annotation, scope/relax `port_security` on the gateway VM port, and clean everything up on deletion. Because the annotation is applied at pod **creation**, existing tenant workloads gain reachability only after they are rolled; the controller reports the pods whose annotation lags the namespace ("pods pending route") rather than restarting anything. +**CNI-mediation controller.** A tenant cannot annotate its own namespace or touch `port_security`, so the platform controller mediates: validate declared remote CIDRs, set the namespace routes annotation, scope/relax `port_security` on the gateway VM port, and clean everything up on deletion. The **remote-CIDR deny-set** a declared CIDR must not overlap is explicit: the cluster **pod / service / join / node** networks (hijacking in-cluster traffic), **`169.254.0.0/16` link-local incl. the cloud metadata endpoint `169.254.169.254`** (credential/SSRF exposure), the platform **LoadBalancer / egress IP pool**, the **host loopback `127.0.0.0/8`**, and the **default route `0.0.0.0/0`** and any prefix so broad it would blackhole cluster traffic (routed mode advertises specific remote networks, never a default route). Cross-tenant overlap between two tenants' *remote* CIDRs is fine — routes are namespace-scoped. A rejected CIDR does not silently no-op: the controller **surfaces the rejection as a `status` condition on the app instance** (`Reason: InvalidRemoteCIDR`, naming the offending CIDR and the network it collides with) and refuses to program the route, so the tenant sees why in the dashboard rather than getting a half-configured gateway. Because the annotation is applied at pod **creation**, existing tenant workloads gain reachability only after they are rolled; the controller reports the pods whose annotation lags the namespace ("pods pending route") rather than restarting anything. -**Security posture.** Routed mode relaxes `port_security` on the gateway port. Containment comes from Cilium's sender-side egress enforcement — the endpoint identity is compiled into the per-endpoint eBPF program, so a spoofed source cannot widen the reachable set beyond the tenant tree + world. The target state is **scoped** `port_security` (the declared remote CIDRs added to the port's allowed addresses). Residual risks are documented in [Security](#security). +**Security posture.** Routed mode adjusts `port_security` on the gateway port so the guest's routed (non-pod-IP) source addresses are not dropped. The shipped state is **scoped** `port_security` — only the declared remote CIDRs are added to the port's allowed addresses, a **Phase 1 acceptance criterion**, not a later hardening step. Defense in depth beyond that comes from Cilium's sender-side egress enforcement — the endpoint identity is compiled into the per-endpoint eBPF program, so even a spoofed source cannot widen the reachable set beyond the tenant tree + world. Residual risks are documented in [Security](#security). **Transport note.** Routed mode carries transparent L3 for TCP/UDP/ICMP/SCTP only — non-standard IP protocols (ESP, GRE, VRRP) still do not cross the pod fabric (same root cause as the ESP finding in [Testing](#testing)). It is kube-ovn-specific and requires the remote CIDR to be disjoint from cluster networks. @@ -128,8 +136,8 @@ Two catalog apps appear in the tenant dashboard, sharing the tunnel/peer/resourc ```yaml # site-gateway (NAT) -## @param tunnel.type {string} Backend: "wireguard" (default; UDP-native) or "ipsec". -tunnel: { type: wireguard } +## @param tunnel.type {string} Backend: "ipsec" (default; broadest interop) or "wireguard" (UDP-native, simpler on an overlay CNI). +tunnel: { type: ipsec } ## @param peer.address {string} Public address of the remote peer / WireGuard endpoint. peer: address: "" @@ -144,8 +152,8 @@ resources: { cpu: "1", memory: "1Gi" } ```yaml # site-router (routed) -## @param tunnel.type {string} Backend: "wireguard" (default) or "ipsec". -tunnel: { type: wireguard } +## @param tunnel.type {string} Backend: "ipsec" (default; broadest interop) or "wireguard" (UDP-native, simpler on an overlay CNI). +tunnel: { type: ipsec } ## @param peer.address {string} Public address of the remote peer / WireGuard endpoint. peer: address: "" @@ -160,6 +168,8 @@ bgp: { enabled: false } resources: { cpu: "1", memory: "1Gi" } ``` +**One peer per instance.** The schema is deliberately **singular `peer:`**, not `peers: []` — each app instance terminates exactly one site-to-site tunnel. A tenant that needs to reach two external sites deploys two instances; the shared foundation makes an instance cheap (one small VM), instances are independently sized/gated/observable, and a blast radius stays scoped to one tunnel. This avoids the breaking-change trap of shipping singular now and retrofitting a list later. Whether a genuine multi-peer single-instance mode is ever warranted (e.g. hub-and-spoke to many sites behind one VM) is deferred as an explicit [open question](#open-questions) rather than pre-committed in the schema. + A cluster admin can enable/disable `site-router` independently — its kube-ovn coupling and `port_security` relaxation may not be wanted on every cluster. No existing app, CRD, or API changes. ## Upgrade and rollback compatibility @@ -170,10 +180,10 @@ Purely additive and opt-in. No migration; existing clusters, manifests, and APIs - **Contained privilege (both apps).** The privileged dataplane lives in a VM guest kernel isolated by hardware virtualization; a compromise inside the gateway cannot manipulate the host node's networking or observe other tenants, and the platform never hands a tenant a privileged host-cluster pod. - **`site-gateway` is fully contained.** SNAT keeps every packet the gateway emits on its own pod IP, so the CNI's port-security never sees a foreign source and needs no relaxation. -- **`site-router` relaxes port_security** on the gateway port (scoped is the target state). Containment is Cilium's sender-side egress enforcement; residual risks to document: intra-namespace identity spoofing (receiver-side identity resolved from source IP) and spoofed-source egress where SNAT is absent. +- **`site-router` scopes port_security** on the gateway port to the declared remote CIDRs (a Phase 1 acceptance criterion — the port is never shipped fully open). Defense in depth is Cilium's sender-side egress enforcement; residual risks to document: intra-namespace identity spoofing (receiver-side identity resolved from source IP) and spoofed-source egress where SNAT is absent. - **Managed apps untouched**; a firewall allow-list on the gateway makes exposure explicit; all state is tenant-scoped. - **Secret handling.** A tenant does not create `Secret` objects directly; as with the existing managed apps (`apps/vpn`, `postgres`, `mariadb`) the app chart templates the `Secret` into the tenant namespace from the app values — autogenerating material when it is not supplied and using `lookup` to keep it stable across reconciles. The design minimizes what must be shipped: for **WireGuard** the tenant's private key is generated in-chart (never entered) and only its public key is surfaced for the remote side, while the peer's public key + endpoint are ordinary non-secret values — so no tenant secret is shipped at all. The one genuinely secret tenant input is the **IPsec PSK** (it must match the remote peer, so it cannot be autogenerated); it rides a values field into the chart-templated `Secret`, with the same plaintext-in-values-at-rest exposure the platform is already addressing for database passwords. The management-API key is generated at runtime, never entered. -- **Management API is platform-only.** The guest's config API is network-restricted so only the platform controller can reach it. +- **Management API is platform-only — enforced outside tenant control.** The guest's config API is the controller's only write path, so the guard against a tenant reaching it directly must be one the tenant *cannot* remove. A NetworkPolicy in the tenant's own namespace is therefore **not** sufficient — the tenant could delete it and then push arbitrary VyOS config, bypassing all controller-side CIDR/target validation. The enforcement must live in the platform's control, not the tenant's: a platform-owned cluster-scoped network policy (e.g. a `CiliumClusterwideNetworkPolicy`) that admits only the controller's identity, and/or binding the management API to an interface/address that is not routable from tenant pods. The **API key is generated at runtime and stored in a platform-controller-namespace Secret — never in a tenant-readable Secret** — so even API reachability does not hand the tenant credentials. This is the one area to **re-verify against the existing production implementation** before Phase 1 acceptance: confirm the shipped guard is genuinely tenant-immutable and the key never lands in the tenant namespace. ## Failure and edge cases @@ -187,7 +197,12 @@ Purely additive and opt-in. No migration; existing clusters, manifests, and APIs ## Testing -The design was **prototyped and validated end-to-end** on a development Cozystack cluster (KubeVirt, Cilium + kube-ovn, LINSTOR), using two gateway VMs — one acting as the tenant gateway, one simulating the external site — with a real tenant-managed Postgres as the inbound target. The **NAT (`site-gateway`) inbound/outbound paths** were validated with the IPsec backend; the **routed (`site-router`) path and the WireGuard backend** are the next to validate (open questions). All items passed: +Validation comes from two independent sources, so **both modes are backed by evidence**: + +- **Routed (`site-router`) — validated in production.** The routed dataplane described here is not speculative: it is the mechanism already running in an existing production Cozystack implementation, deployed for real tenants (and set up by hand for clients before it was productized). The routed mode, its kube-ovn routes annotation, and IPsec termination in a gateway VM are proven in that deployment — which is why Phase 1 leads with it and why it is the primary mode tenants ask for. +- **NAT (`site-gateway`) — prototyped end-to-end.** The NAT paths were **prototyped and validated end-to-end** on a development Cozystack cluster (KubeVirt, Cilium + kube-ovn, LINSTOR), using two gateway VMs — one acting as the tenant gateway, one simulating the external site — with a real tenant-managed Postgres as the inbound target, on the IPsec backend. + +The **WireGuard backend** over the overlay is the remaining item to validate before it is offered as a first-class alternative (see [open questions](#open-questions)). The dev-cluster prototype items below all passed: | # | Item | Result | |---|------|--------| @@ -197,7 +212,7 @@ The design was **prototyped and validated end-to-end** on a development Cozystac | 4 | SNAT-required negative test (removing SNAT black-holes the reply) | PASS | | 5 | MTU / MSS clamp (tunnel MTU 1320, clamp 1280; multi-MB transfer, no stall) | PASS | -**Key finding — native ESP is dropped by the CNI overlay.** IKE (UDP) negotiated and the SA came up, but the ESP data plane was silently dropped even pod-to-pod (receiver saw zero ESP, 100% loss). Forcing ESP-in-UDP encapsulation on both peers restored bidirectional traffic. On an overlay CNI the IPsec backend must therefore always force UDP encapsulation — and **WireGuard, being UDP-native, sidesteps this entirely**, which is a strong argument for it as the default backend. +**Key finding — native ESP is dropped by the CNI overlay.** IKE (UDP) negotiated and the SA came up, but the ESP data plane was silently dropped even pod-to-pod (receiver saw zero ESP, 100% loss). Forcing ESP-in-UDP encapsulation on both peers restored bidirectional traffic. On an overlay CNI the IPsec backend must therefore always force UDP encapsulation — and **WireGuard, being UDP-native, sidesteps this entirely**. This makes WireGuard operationally simpler on the overlay, but it does not change the default: IPsec stays the default for interop breadth and the tenant picks the backend their external site requires (see [Design](#design)). **Non-standard-protocol drop (informs HA).** Independently re-verified with a minimal repro: the CNI drops *all* non-TCP/UDP/ICMP/SCTP IP protocols (proto 112/50/47/4) pod-to-pod, both same-node and cross-node — so it is **not** the geneve tunnel. In policy-enforced tenant pods the dropper is **Cilium's conntrack** (`bpf_lxc.c`, "CT: Unknown L4 protocol"); without a policy it is the OVS datapath. This is the same root cause as the native-ESP drop, and it is why keepalived/VRRP cannot elect over the pod network (see [High availability](#high-availability)). Separately, a kube-ovn `Vip` + `ovn.kubernetes.io/aaps` annotation was validated to share a VIP across two gateway pods with `port_security` kept **on** (the VIP moves on owner change and a bogus source is still dropped — anti-spoofing scoped, not disabled). @@ -209,21 +224,22 @@ Deferred for the first iteration: a **single gateway VM with KubeVirt live-migra ## Rollout -- **Phase 1 — `site-router` (routed), IPsec backend**: the routed dataplane plus its CNI-mediation controller (CIDR validation, routes annotation, scoped `port_security`), with **MSS clamping** and **tunnel-state observability** (SA up/down, rekey, counters) surfaced from the start. Routed is the primary mode tenants ask for. (Note: the end-to-end prototype so far validated the NAT dataplane, so routed carries the first round of validation here.) -- **Phase 2 — `site-gateway` (NAT)**: the inbound/outbound DNAT/SNAT machinery (prototype-validated) with forced UDP encapsulation. -- **Phase 3 — WireGuard backend**: validate over the overlay and, if confirmed simpler/robust, make it the default `tunnel.type`. -- **Phase 4 — the rest**: HA (Service-fronted active/passive), scoped `port_security` for routed, per-tenant egress IP. +- **Phase 1 — `site-router` (routed), IPsec backend** *(blocked on [#29](#scope-and-related-proposals) shipping UDP exposure)*: the routed dataplane plus its CNI-mediation controller (remote-CIDR validation against the full deny-set, routes annotation), with **MSS clamping** and **tunnel-state observability** (SA up/down, rekey, counters) surfaced from the start. **Scoped `port_security` is a Phase 1 acceptance criterion, not a later relaxation** — the gateway port must be scoped to the declared remote CIDRs before Phase 1 is accepted, so the weakest security posture never ships as the released state. This mode is validated in the existing production implementation, which is why it leads. The one thing to resolve *within* this phase is the kube-ovn allowed-address-pairs CIDR support scoped `port_security` needs (see [open questions](#open-questions)); it is a Phase 1 blocker, not a reason to defer scoping. +- **Phase 2 — `site-gateway` (NAT)**: the inbound/outbound DNAT/SNAT machinery (prototype-validated) with forced UDP encapsulation. Fully contained (SNAT), so no `port_security` relaxation at all. +- **Phase 3 — WireGuard backend**: validate the handshake/SA over the overlay and offer it as a **first-class alternative** backend. **The default stays IPsec** — the tenant picks the backend their site requires, so WireGuard is added alongside IPsec, never promoted over it. +- **Phase 4 — the rest**: HA (Service-fronted active/passive), per-tenant egress IP. -Ordering leads with routed because it is the primary requested mode; the shared foundation is built once and both apps sit on it. +Ordering leads with routed because it is the primary requested mode and the one already proven in production; the shared foundation is built once and both apps sit on it. Scoped `port_security` and MSS clamping are in Phase 1 by design so the first release is not the least-secure one. ## Open questions - **IPsec PSK at rest.** WireGuard ships no tenant secret (its key is autogenerated in-chart); the IPsec PSK is the one tenant-supplied secret — decide whether it stays a values field templated into a `Secret` (like DB passwords today — plaintext in values at rest) or gets a write-only / dashboard-mediated affordance. -- **Tunnel exposure via #29.** Confirm the `ExposureClass` UDP support and how the tunnel endpoint address interacts with the tenant's LB pool and quotas. +- **Tunnel exposure via #29 (blocking).** #29 with **UDP** `ExposureClass` support is a declared Phase 1 blocker (see [Scope](#scope-and-related-proposals)); the open detail is how the tunnel endpoint address interacts with the tenant's LB pool and quotas, and whether the stopgap bespoke UDP `LoadBalancer` is needed if #29 slips. - **Topology-advertising apps under NAT.** Whether per-member advertised addressing for Kafka / Mongo / Redis-Cluster / Cassandra under `site-gateway` is worth a tunnel-backed `ExposureClass` (needs per-member granularity #29 does not have today) or is simply left to `site-router`. - **HA mechanism.** Given VRRP does not cross the overlay (see Testing), standardize on Service/endpoint-based failover (needs a controller/lease) vs an AAP shared-VIP with side-channel election. -- **Scoped port_security for `site-router`.** kube-ovn allowed-address-pairs need CIDR support to add the declared remote CIDRs to the port (OVN itself accepts `MAC IP/mask`). -- **WireGuard backend.** Validate the handshake/SA over the overlay and confirm identical bridging before defaulting to it. +- **Scoped port_security for `site-router` (Phase 1 blocker).** Scoped `port_security` is a Phase 1 acceptance criterion (see [Rollout](#rollout)), so this must be resolved *within* Phase 1, not deferred: kube-ovn allowed-address-pairs need CIDR support to add the declared remote CIDRs to the port (OVN itself accepts `MAC IP/mask`). Confirm how the existing production implementation scopes the port today and reuse that approach. +- **WireGuard backend.** Validate the handshake/SA over the overlay and confirm identical bridging before offering it as a first-class alternative. This does **not** gate on becoming the default — the default stays IPsec regardless; WireGuard is added alongside it (see [Rollout](#rollout)). +- **Multi-peer per instance.** The schema is singular `peer:` and the current position is **one tunnel per app instance** — a tenant reaching two sites deploys two instances. This needs verification before it is locked in: (a) *does anyone actually need* a single VM terminating many peers, or does per-instance always suffice? The working assumption is per-instance suffices (a new instance is cheap and independently gated/sized/observed), and it avoids a breaking `peer: → peers: []` migration later. (b) *pros/cons* — one hub VM would share the tunnel-drop blast radius and a single throughput envelope across all peers, whereas per-instance isolates them but costs one VM each; hub-and-spoke to many small sites is the only clear case where one VM might win. (c) *how the existing production implementation models this today* — confirm it is single-peer-per-instance and that no deployed tenant relies on multi-peer, so committing to singular is not a regression. If multi-peer is ever needed it would be an additive `peers: []` on a *new* schema version, not a retrofit of this one. - **Relationship to ClusterMesh (#7).** Whether to present these apps and ClusterMesh as one coordinated "tenant connectivity" story, and where the boundary sits. ## Alternatives considered From 8bad3a6dbbd1fad9973e370f9cfb5c2d0b4ccd53 Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Thu, 16 Jul 2026 17:06:20 +0500 Subject: [PATCH 5/5] docs: clarify site-router port-security risks Signed-off-by: Myasnikov Daniil --- .../tenant-site-connectivity/README.md | 23 +++++++++++-------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/design-proposals/tenant-site-connectivity/README.md b/design-proposals/tenant-site-connectivity/README.md index 4342553..80fb320 100644 --- a/design-proposals/tenant-site-connectivity/README.md +++ b/design-proposals/tenant-site-connectivity/README.md @@ -65,7 +65,7 @@ Both apps share roughly 80% of their implementation — the VM appliance, the tu | Source IP | not preserved (SNAT) | preserved | | Reachability | per host:port | whole subnet + ICMP | | Config surface | `exposedServices` / `remoteTargets` (per-target ports) | `remoteCIDRs` / static routes / BGP | -| Security posture | fully contained | port_security scoped to declared remote CIDRs (Phase 1 acceptance) | +| Security posture | fully contained | gateway port security disabled; Cilium destination policy + guest firewall required | | Topology-advertising apps | need per-member advertised addresses | transparent | Why two apps rather than one `mode:` toggle: the two have divergent values schemas (per-target ports vs CIDRs/BGP), asymmetric controllers (routed needs a privileged CNI-mediation controller; NAT needs none), and different security postures a cluster admin may want to gate independently. Sharing the implementation (a common VyOS-driving controller core + image + base chart) keeps this from becoming duplicated code. @@ -124,9 +124,9 @@ VyOS has no free LTS channel — only the rolling release is freely buildable The decrypted packet is forwarded onto the tenant network **without NAT**; the return path is the kube-ovn `ovn.kubernetes.io/routes` namespace annotation (inherited onto pods at creation by the kube-ovn webhook), so the **client source IP is preserved** and whole-subnet reachability + ICMP work. This is the mode for cases where the remote side authorizes by source IP, or needs to reach a whole subnet, or where the app advertises its own topology (members' real addresses are directly reachable). -**CNI-mediation controller.** A tenant cannot annotate its own namespace or touch `port_security`, so the platform controller mediates: validate declared remote CIDRs, set the namespace routes annotation, scope/relax `port_security` on the gateway VM port, and clean everything up on deletion. The **remote-CIDR deny-set** a declared CIDR must not overlap is explicit: the cluster **pod / service / join / node** networks (hijacking in-cluster traffic), **`169.254.0.0/16` link-local incl. the cloud metadata endpoint `169.254.169.254`** (credential/SSRF exposure), the platform **LoadBalancer / egress IP pool**, the **host loopback `127.0.0.0/8`**, and the **default route `0.0.0.0/0`** and any prefix so broad it would blackhole cluster traffic (routed mode advertises specific remote networks, never a default route). Cross-tenant overlap between two tenants' *remote* CIDRs is fine — routes are namespace-scoped. A rejected CIDR does not silently no-op: the controller **surfaces the rejection as a `status` condition on the app instance** (`Reason: InvalidRemoteCIDR`, naming the offending CIDR and the network it collides with) and refuses to program the route, so the tenant sees why in the dashboard rather than getting a half-configured gateway. Because the annotation is applied at pod **creation**, existing tenant workloads gain reachability only after they are rolled; the controller reports the pods whose annotation lags the namespace ("pods pending route") rather than restarting anything. +**CNI-mediation controller.** A tenant cannot annotate its own namespace or touch `port_security`, so the platform controller mediates: validate declared remote CIDRs, set the namespace routes annotation, disable `port_security` only on the gateway VM port, and restore the normal setting on deletion. The **remote-CIDR deny-set** a declared CIDR must not overlap is explicit: the cluster **pod / service / join / node** networks (hijacking in-cluster traffic), **`169.254.0.0/16` link-local incl. the cloud metadata endpoint `169.254.169.254`** (credential/SSRF exposure), the platform **LoadBalancer / egress IP pool**, the **host loopback `127.0.0.0/8`**, and the **default route `0.0.0.0/0`** and any prefix so broad it would blackhole cluster traffic (routed mode advertises specific remote networks, never a default route). Cross-tenant overlap between two tenants' *remote* CIDRs is fine — routes are namespace-scoped. A rejected CIDR does not silently no-op: the controller **surfaces the rejection as a `status` condition on the app instance** (`Reason: InvalidRemoteCIDR`, naming the offending CIDR and the network it collides with) and refuses to program the route, so the tenant sees why in the dashboard rather than getting a half-configured gateway. This validation constrains the routes programmed by the controller; it does **not** prove that packets arriving from the remote peer use a declared source. The controller therefore also installs a platform-owned VyOS tunnel-ingress firewall rule that drops packets whose source is outside `remoteCIDRs`, and does not report the instance Ready until that rule is active. Because the namespace routes annotation is applied at pod **creation**, existing tenant workloads gain reachability only after they are rolled; the controller reports the pods whose annotation lags the namespace ("pods pending route") rather than restarting anything. -**Security posture.** Routed mode adjusts `port_security` on the gateway port so the guest's routed (non-pod-IP) source addresses are not dropped. The shipped state is **scoped** `port_security` — only the declared remote CIDRs are added to the port's allowed addresses, a **Phase 1 acceptance criterion**, not a later hardening step. Defense in depth beyond that comes from Cilium's sender-side egress enforcement — the endpoint identity is compiled into the per-endpoint eBPF program, so even a spoofed source cannot widen the reachable set beyond the tenant tree + world. Residual risks are documented in [Security](#security). +**Security posture.** Routed mode disables `port_security` on the gateway port so the guest's routed (non-pod-IP) source addresses are not dropped. kube-ovn v1.15.10 cannot express a CIDR in its allowed-address-pair path: it accepts host IPs only, even though OVN itself supports `MAC IP/mask`. Phase 1 therefore ships with the gateway logical port fully relaxed and treats scoped port security as follow-up hardening. This means OVN no longer validates the source address of packets emitted by the gateway. Cilium still applies sender-side egress policy using the gateway endpoint's compiled identity, independent of a packet's claimed source IP. That is expected to block direct access to unrelated tenant endpoints, nodes, and the Kubernetes API, but it limits **destinations**, not source authenticity: `world`, same-tenant descendants, and explicitly allowed platform services remain reachable. The guest source filter above and the additional controls in [Security](#security) are consequently part of the Phase 1 boundary, not optional defense in depth. **Transport note.** Routed mode carries transparent L3 for TCP/UDP/ICMP/SCTP only — non-standard IP protocols (ESP, GRE, VRRP) still do not cross the pod fabric (same root cause as the ESP finding in [Testing](#testing)). It is kube-ovn-specific and requires the remote CIDR to be disjoint from cluster networks. @@ -174,16 +174,17 @@ A cluster admin can enable/disable `site-router` independently — its kube-ovn ## Upgrade and rollback compatibility -Purely additive and opt-in. No migration; existing clusters, manifests, and APIs are unaffected. The gateway VM image is a new artifact built and published by the platform. Rollback is deletion of the app instance (its `VMInstance`/`VMDisk`, the generated Services, and — for `site-router` — the namespace annotation / port_security scoping the controller cleans up), which removes the gateway entirely; nothing else in the tenant is touched. +Purely additive and opt-in. No migration; existing clusters, manifests, and APIs are unaffected. The gateway VM image is a new artifact built and published by the platform. Rollback is deletion of the app instance (its `VMInstance`/`VMDisk`, the generated Services, and — for `site-router` — the namespace annotation / port-security relaxation the controller restores), which removes the gateway entirely; nothing else in the tenant is touched. ## Security - **Contained privilege (both apps).** The privileged dataplane lives in a VM guest kernel isolated by hardware virtualization; a compromise inside the gateway cannot manipulate the host node's networking or observe other tenants, and the platform never hands a tenant a privileged host-cluster pod. - **`site-gateway` is fully contained.** SNAT keeps every packet the gateway emits on its own pod IP, so the CNI's port-security never sees a foreign source and needs no relaxation. -- **`site-router` scopes port_security** on the gateway port to the declared remote CIDRs (a Phase 1 acceptance criterion — the port is never shipped fully open). Defense in depth is Cilium's sender-side egress enforcement; residual risks to document: intra-namespace identity spoofing (receiver-side identity resolved from source IP) and spoofed-source egress where SNAT is absent. -- **Managed apps untouched**; a firewall allow-list on the gateway makes exposure explicit; all state is tenant-scoped. +- **`site-router` disables port security on one logical port.** An empty OVN `port_security` list permits arbitrary source addresses from the gateway. Cozystack also disables Cilium source-IP verification in its chained-CNI mode, so a compromised or misconfigured remote peer can inject packets with undeclared source addresses unless VyOS rejects them. The tunnel-ingress source allow-list is therefore mandatory and platform-owned; controller-side `remoteCIDRs` validation alone is not a data-plane guard. +- **Cilium contains destinations, not identities claimed in packets.** Sender-side egress enforcement uses the gateway endpoint's compiled identity, so source spoofing should not grant direct access to unrelated tenants, nodes, or the Kubernetes API. It does not make the relaxed port safe by itself: the tenant policy permits `world`, the tenant tree, and selected platform services. A malicious peer could otherwise send spoofed-source traffic to external destinations, attempt link-local metadata access where routing permits it, attack allowed platform services, or reach other tenants through their intentionally public endpoints. The gateway gets a platform-owned Cilium `egressDeny` for `169.254.0.0/16` and concrete management ranges that do not overlap tenant workloads, while its guest forwarding policy denies tunnel-to-`world` traffic unless a future explicit use case enables it. +- **Managed apps untouched.** The guest firewall makes exposure explicit, defaults tunnel forwarding to deny, admits only declared remote sources and intended tenant destinations, and remains entirely platform-controlled. - **Secret handling.** A tenant does not create `Secret` objects directly; as with the existing managed apps (`apps/vpn`, `postgres`, `mariadb`) the app chart templates the `Secret` into the tenant namespace from the app values — autogenerating material when it is not supplied and using `lookup` to keep it stable across reconciles. The design minimizes what must be shipped: for **WireGuard** the tenant's private key is generated in-chart (never entered) and only its public key is surfaced for the remote side, while the peer's public key + endpoint are ordinary non-secret values — so no tenant secret is shipped at all. The one genuinely secret tenant input is the **IPsec PSK** (it must match the remote peer, so it cannot be autogenerated); it rides a values field into the chart-templated `Secret`, with the same plaintext-in-values-at-rest exposure the platform is already addressing for database passwords. The management-API key is generated at runtime, never entered. -- **Management API is platform-only — enforced outside tenant control.** The guest's config API is the controller's only write path, so the guard against a tenant reaching it directly must be one the tenant *cannot* remove. A NetworkPolicy in the tenant's own namespace is therefore **not** sufficient — the tenant could delete it and then push arbitrary VyOS config, bypassing all controller-side CIDR/target validation. The enforcement must live in the platform's control, not the tenant's: a platform-owned cluster-scoped network policy (e.g. a `CiliumClusterwideNetworkPolicy`) that admits only the controller's identity, and/or binding the management API to an interface/address that is not routable from tenant pods. The **API key is generated at runtime and stored in a platform-controller-namespace Secret — never in a tenant-readable Secret** — so even API reachability does not hand the tenant credentials. This is the one area to **re-verify against the existing production implementation** before Phase 1 acceptance: confirm the shipped guard is genuinely tenant-immutable and the key never lands in the tenant namespace. +- **Management API is platform-only — enforced at both network boundaries.** The guest's config API is the controller's only write path, so the guard against a tenant or tunnel peer reaching it must be one they cannot remove. An allow-only `CiliumClusterwideNetworkPolicy` is insufficient on its own: Cilium allow rules are additive with the tenant's existing broad ingress policy, and a packet decrypted by VyOS and addressed to the guest itself does not cross the pod veth where Cilium enforces policy. The API therefore binds only to the pod-network address (never a tunnel address), and a platform-owned VyOS local-input policy drops its port on every tunnel interface. At the pod boundary, the base tenant policy excludes the gateway endpoint and a platform-owned gateway policy admits only the public tunnel UDP ports plus the controller's identity on the API port. A separate management interface/VRF is valid additional hardening, but is not required for this two-boundary enforcement model. The **API key is generated at runtime and stored in a platform-controller-namespace Secret — never in a tenant-readable Secret** — so reachability alone does not hand the tenant credentials. Phase 1 verifies all of these properties against the implemented datapath. ## Failure and edge cases @@ -218,18 +219,20 @@ The **WireGuard backend** over the overlay is the remaining item to validate bef Planned automated coverage before implementation lands: helm-unittest for chart rendering across backends and both apps; an e2e that stands up the two-VM topology and asserts the inbound/outbound flows plus the SNAT-required and MSS behaviors. +Phase 1 also has explicit negative security acceptance tests. From the remote side, packets with a source outside `remoteCIDRs`, packets to another tenant's pod or `ClusterIP`, node and Kubernetes API addresses, `169.254.0.0/16`, the VyOS management API, and arbitrary `world` destinations must all be dropped. A declared remote source reaching an intended same-tenant destination must pass with its source preserved. Tests cover both direct cluster addresses and public re-entry paths, and use Hubble plus guest counters to establish which guard performed each drop. + ## High availability Deferred for the first iteration: a **single gateway VM with KubeVirt live-migration** for planned maintenance (conntrack, tunnel SA, and pod IP preserved — no tunnel drop; requires migratable/replicated storage, see the `blockSize` caveat). Automatic **unplanned-failure** HA is a follow-up; the preferred path is **Service-fronted active/passive** because it is CNI-agnostic (no floating L2 VIP, so no port-security interaction) at the cost of a small leader-election/lease agent and seconds-scale endpoint reconvergence. A kube-ovn shared-VIP via allowed-address-pairs was validated to move a VIP with `port_security` kept on, but keepalived-style election cannot run over the pod network (VRRP/proto-112 is dropped — see [Testing](#testing)), so it would need a side-channel or controller to drive the VIP claim. ## Rollout -- **Phase 1 — `site-router` (routed), IPsec backend** *(blocked on [#29](#scope-and-related-proposals) shipping UDP exposure)*: the routed dataplane plus its CNI-mediation controller (remote-CIDR validation against the full deny-set, routes annotation), with **MSS clamping** and **tunnel-state observability** (SA up/down, rekey, counters) surfaced from the start. **Scoped `port_security` is a Phase 1 acceptance criterion, not a later relaxation** — the gateway port must be scoped to the declared remote CIDRs before Phase 1 is accepted, so the weakest security posture never ships as the released state. This mode is validated in the existing production implementation, which is why it leads. The one thing to resolve *within* this phase is the kube-ovn allowed-address-pairs CIDR support scoped `port_security` needs (see [open questions](#open-questions)); it is a Phase 1 blocker, not a reason to defer scoping. +- **Phase 1 — `site-router` (routed), IPsec backend** *(blocked on [#29](#scope-and-related-proposals) shipping UDP exposure)*: the routed dataplane plus its CNI-mediation controller (remote-CIDR validation against the full deny-set, routes annotation, gateway-only `port_security` relaxation), with **MSS clamping**, **tunnel-state observability** (SA up/down, rekey, counters), and the source, destination, metadata, public-forwarding, and management-API guards described in [Security](#security). Full relaxation is accepted because kube-ovn v1.15.10 cannot emit CIDR-scoped allowed addresses; the resulting arbitrary-source injection risk is explicit, and the source allow-list is enforced in the guest. Phase 1 is not accepted until the negative security tests pass. This mode is validated in the existing production implementation, which is why it leads. - **Phase 2 — `site-gateway` (NAT)**: the inbound/outbound DNAT/SNAT machinery (prototype-validated) with forced UDP encapsulation. Fully contained (SNAT), so no `port_security` relaxation at all. - **Phase 3 — WireGuard backend**: validate the handshake/SA over the overlay and offer it as a **first-class alternative** backend. **The default stays IPsec** — the tenant picks the backend their site requires, so WireGuard is added alongside IPsec, never promoted over it. - **Phase 4 — the rest**: HA (Service-fronted active/passive), per-tenant egress IP. -Ordering leads with routed because it is the primary requested mode and the one already proven in production; the shared foundation is built once and both apps sit on it. Scoped `port_security` and MSS clamping are in Phase 1 by design so the first release is not the least-secure one. +Ordering leads with routed because it is the primary requested mode and the one already proven in production; the shared foundation is built once and both apps sit on it. MSS clamping, guest source filtering, management isolation, gateway-specific Cilium denies, and the negative security suite ship in Phase 1 alongside the necessary port-security relaxation. ## Open questions @@ -237,7 +240,7 @@ Ordering leads with routed because it is the primary requested mode and the one - **Tunnel exposure via #29 (blocking).** #29 with **UDP** `ExposureClass` support is a declared Phase 1 blocker (see [Scope](#scope-and-related-proposals)); the open detail is how the tunnel endpoint address interacts with the tenant's LB pool and quotas, and whether the stopgap bespoke UDP `LoadBalancer` is needed if #29 slips. - **Topology-advertising apps under NAT.** Whether per-member advertised addressing for Kafka / Mongo / Redis-Cluster / Cassandra under `site-gateway` is worth a tunnel-backed `ExposureClass` (needs per-member granularity #29 does not have today) or is simply left to `site-router`. - **HA mechanism.** Given VRRP does not cross the overlay (see Testing), standardize on Service/endpoint-based failover (needs a controller/lease) vs an AAP shared-VIP with side-channel election. -- **Scoped port_security for `site-router` (Phase 1 blocker).** Scoped `port_security` is a Phase 1 acceptance criterion (see [Rollout](#rollout)), so this must be resolved *within* Phase 1, not deferred: kube-ovn allowed-address-pairs need CIDR support to add the declared remote CIDRs to the port (OVN itself accepts `MAC IP/mask`). Confirm how the existing production implementation scopes the port today and reuse that approach. +- **Scoped port security for `site-router` (follow-up hardening).** kube-ovn v1.15.10 allowed-address-pairs accept host IPs only, while OVN itself accepts `MAC IP/mask`; Phase 1 therefore cannot scope a whole routed prefix through the supported kube-ovn API. Track upstream CIDR support (or contribute it), then replace full relaxation with declared-prefix scoping without removing the guest source filter or negative tests. - **WireGuard backend.** Validate the handshake/SA over the overlay and confirm identical bridging before offering it as a first-class alternative. This does **not** gate on becoming the default — the default stays IPsec regardless; WireGuard is added alongside it (see [Rollout](#rollout)). - **Multi-peer per instance.** The schema is singular `peer:` and the current position is **one tunnel per app instance** — a tenant reaching two sites deploys two instances. This needs verification before it is locked in: (a) *does anyone actually need* a single VM terminating many peers, or does per-instance always suffice? The working assumption is per-instance suffices (a new instance is cheap and independently gated/sized/observed), and it avoids a breaking `peer: → peers: []` migration later. (b) *pros/cons* — one hub VM would share the tunnel-drop blast radius and a single throughput envelope across all peers, whereas per-instance isolates them but costs one VM each; hub-and-spoke to many small sites is the only clear case where one VM might win. (c) *how the existing production implementation models this today* — confirm it is single-peer-per-instance and that no deployed tenant relies on multi-peer, so committing to singular is not a regression. If multi-peer is ever needed it would be an additive `peers: []` on a *new* schema version, not a retrofit of this one. - **Relationship to ClusterMesh (#7).** Whether to present these apps and ClusterMesh as one coordinated "tenant connectivity" story, and where the boundary sits.