design-proposal: tenant-managed site-to-site connectivity via gateway VMs (site-router + site-gateway)#30
Conversation
Adds a design proposal for tenant-managed site-to-site connectivity (IPsec or WireGuard) terminated in a KubeVirt VM that NAT-bridges an external site to the tenant's managed-app ClusterIPs, both directions, without granting tenants privileged host-cluster pods. Includes the end-to-end prototype validation results and the relationship to the ClusterMesh/Kilo proposal (#7). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughChangesTenant Site Connectivity Proposal
Estimated code review effort: 2 (Simple) | ~10 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a design proposal for tenant-managed external network connectivity via a gateway VM (IPsec / WireGuard) in Cozystack, which terminates VPN tunnels inside a KubeVirt VM to isolate privileged networking. The review feedback highlights several key technical refinements for the proposal: ensuring the guest OS is configured to own virtual service-exposure addresses for inbound traffic, addressing potential port collisions for outbound traffic to multiple remote targets by mapping them to unique targetPorts, providing concrete KubeVirt API examples for the block size override, and noting that standard VRRP for high availability may be blocked by CNI-level IP/MAC spoofing prevention, requiring alternative failover mechanisms.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
|
|
||
| ### Inbound path (external site → any tenant app) | ||
|
|
||
| The remote peer targets a virtual "service-exposure" address that the gateway owns and maps to an app `ClusterIP`. |
There was a problem hiding this comment.
For the inbound path, the remote peer targets a virtual "service-exposure" address. To ensure the gateway VM can receive and DNAT this traffic, the guest OS needs to configure these virtual IPs (e.g., as alias IPs or on a dummy interface) so that the guest kernel accepts packets destined for them.
| The remote peer targets a virtual "service-exposure" address that the gateway owns and maps to an app `ClusterIP`. | |
| The remote peer targets a virtual "service-exposure" address that the gateway VM owns (configured as an alias or dummy interface IP inside the guest OS via cloud-init) and maps to an app ClusterIP. |
There was a problem hiding this comment.
This isn't needed for the validated design: the tunnel is route-based (VTI), so the decrypted packet arrives on the tunnel interface and is DNAT'd in PREROUTING — which runs before the kernel's local-delivery check — so the gateway need not own the virtual service-exposure address on an interface. The dev-cluster prototype confirmed this: nat destination rule inbound-interface vti0 destination <vip> → <ClusterIP> worked with no alias/dummy IP assigned. Keeping the text as-is to avoid documenting a requirement that doesn't apply to the route-based path.
| 1. The app connects to a local Service name the chart creates (e.g. `remote-db.<tenant>.svc:port`) — the only app-side change is the target hostname. | ||
| 2. The node routes the `ClusterIP` to the gateway VM. | ||
| 3. **DNAT**: the local listener → the real remote `IP:port` behind the tunnel; the VM encrypts and sends it over the tunnel; SNAT into the tunnel's inner subnet lets replies return. |
There was a problem hiding this comment.
When using a local ClusterIP Service to route outbound traffic to the gateway VM, the CNI/kube-proxy on the client node will perform DNAT, translating the Service's ClusterIP to the gateway VM's pod IP before the packet reaches the VM.\n\nIf a tenant has multiple remote targets that use the same destination port (e.g., two different remote databases on port 5432), the gateway VM will receive packets for both targets with the same destination IP (the VM's pod IP) and destination port (5432). It will have no way to distinguish which remote target was originally intended.\n\nTo resolve this, the design should explicitly specify that each remote target must map to a unique targetPort (local listener port) on the gateway VM. The local Services can still expose the standard port (e.g., 5432), but they must map to these unique targetPorts on the gateway VM (e.g., 5432 and 5433), which the gateway VM then DNATs to the respective remote IP:port.
| 1. The app connects to a local Service name the chart creates (e.g. `remote-db.<tenant>.svc:port`) — the only app-side change is the target hostname. | |
| 2. The node routes the `ClusterIP` to the gateway VM. | |
| 3. **DNAT**: the local listener → the real remote `IP:port` behind the tunnel; the VM encrypts and sends it over the tunnel; SNAT into the tunnel's inner subnet lets replies return. | |
| 1. The app connects to a local Service name the chart creates (e.g. remote-db.<tenant>.svc:port) — the only app-side change is the target hostname.\n2. The node translates the ClusterIP to the gateway VM's pod IP and a unique targetPort (local listener port) allocated on the VM. This unique port is required so the VM can distinguish between multiple remote targets that use the same destination port.\n3. **DNAT**: the unique local listener port on the VM -> the real remote IP:port behind the tunnel; the VM encrypts and sends it over the tunnel; SNAT into the tunnel's inner subnet lets replies return. |
There was a problem hiding this comment.
Good catch — addressed in 72db47e. The outbound path now states each remote target maps to a per-target-unique listener port on the gateway (the local Service still exposes the standard port, e.g. 5432), so the VM can disambiguate multiple targets sharing a destination port. The values schema's remoteTargets[].localPort already carries this.
| - **Native ESP dropped by the CNI overlay (IPsec).** On Cilium/kube-ovn, native ESP (IP proto 50) does not traverse the overlay even pod-to-pod; ESP-in-UDP (forced UDP encapsulation) is required unconditionally (validated — see Testing). WireGuard, being UDP-native, is unaffected. | ||
| - **DNAT must target stable `ClusterIP`s**, never ephemeral pod IPs. | ||
| - **Source IP is lost inbound** (SNAT) — only relevant for apps with source-IP ACLs. | ||
| - **VM image must be bootable and 512-native.** DRBD-backed (4K-sector) StorageClasses cannot boot a 512-native GPT image; the disk must sit on a 512-native StorageClass or use a KubeVirt `blockSize` override. The image must ship a real bootloader (validated the hard way — see Testing). |
There was a problem hiding this comment.
To make the KubeVirt blockSize override actionable and clear for the implementer, it would be beneficial to include a brief example of the exact KubeVirt API fields used to configure it.
| - **VM image must be bootable and 512-native.** DRBD-backed (4K-sector) StorageClasses cannot boot a 512-native GPT image; the disk must sit on a 512-native StorageClass or use a KubeVirt `blockSize` override. The image must ship a real bootloader (validated the hard way — see Testing). | |
| - **VM image must be bootable and 512-native.** DRBD-backed (4K-sector) StorageClasses cannot boot a 512-native GPT image; the disk must sit on a 512-native StorageClass or use a KubeVirt blockSize override (e.g., setting blockSize.custom.logical to 512 and blockSize.custom.physical to 4096 in the disk spec). The image must ship a real bootloader (validated the hard way — see Testing). |
There was a problem hiding this comment.
Done in 72db47e — added the concrete example (blockSize.custom.logical: 512 / physical: 4096 on the disk spec).
|
|
||
| - **Secret handling.** Best shape for mounting PSK/cert/WireGuard-key material into the guest without persisting it in plaintext cloud-init at rest. | ||
| - **VIP allocation.** One LoadBalancer VIP per gateway vs. sharing; interaction with the tenant's LB address pool and quotas. | ||
| - **HA.** Is active/standby (e.g. VRRP + shared VIP) in scope for a later iteration? A single VM is a per-tenant SPOF. |
There was a problem hiding this comment.
In standard Kubernetes CNIs (including Cilium and kube-ovn), IP/MAC spoofing prevention is typically enabled by default. This means standard VRRP (keepalived) running inside guest VMs on the default pod network may fail because the CNI will block gratuitous ARPs or refuse to route the shared VIP to the standby VM.\n\nTo support HA, the platform might need to integrate with CNI-specific virtual IP features or use a Kubernetes Service/EndpointSlice-based failover mechanism instead of pure guest-level VRRP.
| - **HA.** Is active/standby (e.g. VRRP + shared VIP) in scope for a later iteration? A single VM is a per-tenant SPOF. | |
| - **HA.** Is active/standby in scope for a later iteration? Note that standard VRRP (e.g., keepalived with a shared VIP) inside guest VMs on the default pod network may fail due to CNI-level IP/MAC spoofing prevention (which blocks gratuitous ARPs and unauthorized IPs). The HA design will likely need to integrate with CNI-specific virtual IP features or use a Kubernetes Service/EndpointSlice-based failover mechanism. |
There was a problem hiding this comment.
Agreed that guest-level VRRP with a shared VIP is problematic — the new High Availability section covers it (72db47e). One correction from validation: the blocker isn't IP/MAC anti-spoofing (that's solvable with a scoped kube-ovn allowed-address-pair — validated: the VIP moves through OVN with port-security kept on). The real blocker is that VRRP advertisements (IP proto 112) are dropped pod-to-pod by Cilium's conntrack (non-TCP/UDP/ICMP/SCTP; see Cilium CFP #39601, closed not-planned), so keepalived can't elect over the pod network. HA therefore uses Service/endpoint failover or a side-channel for the election, as documented.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@design-proposals/tenant-site-gateway/README.md`:
- Around line 157-158: The tenant-secret flow is still underspecified in the
README: define the concrete delivery path for PSKs/certs/WireGuard keys into the
guest without putting them in the VM spec or cloud-init, and document the
post-reconciliation read-access model clearly. Update the tenant-supplied
secrets section to name the component or mechanism that performs delivery and
the expected access boundaries so the security story is explicit.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: f6ce1e74-edfe-4394-9c0e-b857bcb66a81
📒 Files selected for processing (1)
design-proposals/tenant-site-gateway/README.md
Adds a High availability section (KubeVirt live-migration for planned maintenance; Service-fronted active/passive and kube-ovn allowed-address-pairs shared-VIP for unplanned failure), with the shared-VIP mechanism validated on a development cluster. Attributes the pod-to-pod drop of non-TCP/UDP/ICMP/SCTP IP protocols (VRRP proto 112, ESP proto 50) to Cilium's conntrack rather than the geneve tunnel, unifying it with the native-ESP finding. Also addresses review feedback: per-target-unique outbound listener ports, a concrete blockSize example, a note that SNAT keeps the gateway anti-spoofing clean, and a stronger secret-handling open question. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Andrei Kvapil (kvaps)
left a comment
Andrei Kvapil (kvaps)
left a comment
There was a problem hiding this comment.
Thanks — the e2e validation here (the ESP-in-UDP finding, the SNAT negative test, the AAP scoping proof) is exactly the groundwork we needed. We discussed the direction internally; three structural changes before this is ready, plus nits.
1. Routed (no-NAT) mode is the primary mechanism and belongs in Phase 1.
Tenants already run this pattern today (virtual routers: the ovn.kubernetes.io/routes namespace annotation + relaxed port security), so shipping NAT-first would regress their UX. Please restructure around two modes:
- routed (default): the routes annotation (kubeovn-webhook already inherits it onto pods at creation) provides the return path, so SNAT is unnecessary and the client source IP is preserved. The "missing SNAT → black-hole" finding holds only for NAT mode — scope it accordingly.
- NAT (fallback): keep the current design as an enumerated-targets mode for when routing is impossible (remote CIDR overlaps podCIDR/serviceCIDR/node networks) or the remote side only accepts narrow traffic selectors. State the limits honestly: per-host:port entries, no whole-subnet reachability, no ICMP. NETMAP-style 1:1 subnet mapping → future work.
Anti-spoofing for routed mode: disabling port_security on the gateway VM port is acceptable for v1. Containment comes from Cilium's sender-side egress enforcement — the endpoint identity is compiled into the per-endpoint eBPF program on the node-side veth, so the policy verdict doesn't depend on the packet's source IP; a spoofed source cannot widen the reachable set beyond the tenant tree + world. Document the residual risks in Security: identity spoofing within the tenant's own namespace (receiver-side identity is resolved from src IP via ipcache), and spoofed-source egress where SNAT is absent. Target state (follow-up, not v1): scoped port_security — keep it enabled and add the declared remote CIDRs to the port's allowed addresses; your AAP validation is the right mechanism, it only lacks CIDR support in kube-ovn (OVN itself accepts MAC IP/mask).
Also state explicitly: routed mode is transparent L3 for TCP/UDP/ICMP/SCTP only — per your own conntrack finding, ESP/GRE/OSPF/VRRP still won't cross the pod fabric.
2. Day-2 reconfiguration needs a section, with the mechanism agreed now.
Direction we settled on: VyOS HTTP API driven by a platform controller — render the target config from values and push it. VyOS commits are transactional, so unrelated changes don't touch the tunnel and key rotation only re-establishes the SA. The API key is delivered once at first boot via cloud-init; the API must be reachable by the platform only (define that boundary in Security). Implementation may land as a separate phase — v1 may temporarily fall back to "config change = VM reboot" if called out explicitly.
3. Add a "low-level entities & controller" section.
Tenants can't (and shouldn't) annotate their namespaces, so a platform controller mediates:
- user input stays in the catalog app values — no new CRD (the aggregated API server just mirrors HelmReleases); validation = values.schema plus controller-side checks at reconcile, surfaced as status conditions;
- the controller validates declared remote CIDRs (disjoint from podCIDR/serviceCIDR/join/node networks; overlap between tenants is fine — routes are namespace-scoped), sets the namespace routes annotation, relaxes/scopes port security on the gateway VM port, and cleans everything up on app deletion;
- routes apply only to newly created pods (the webhook mutates at Create) — no auto-restart of tenant workloads; the controller reports pods whose annotation lags the namespace ("pods pending route") and the user rolls them;
- keep the CNI-touching operations behind a small backend interface so a future CNI can implement them natively.
Nits:
- Inbound exposure model is inconsistent: the text uses virtual IPs (
10.200.0.10:5432), the values sketch uses{name, listenPort, targetService, port}— pick one. In routed mode the remote side hits ClusterIPs directly, so virtual exposure addresses are a NAT-mode-only concept. - HA: agreed with your recommendation (single VM + live-migration in v1). For the follow-up, prefer the Service-fronted active/passive path — CNI-agnostic, which matters since we don't want to deepen kube-ovn coupling.
- Sync with the portal-side implementation before building the image — it uses the same VyOS + cloud-init shape; image and config format should be shared.
- The tunnel LoadBalancer should align with the structured external exposure redesign (#29) rather than adding another ad-hoc LB path.
- Scope: list per-tenant egress IP as explicitly deferred future work — the gateway VM is its natural home later.
|
Thanks for the thorough review — this is the right shape and most of it I'm taking as-is. Agreed on all three structural changes (two explicit modes, a day-2 section, a low-level-entities/controller section) and the nits. One adjustment to the mode ordering, reasoning below; everything else lands as you described. 1. Modes: agreed — with NAT as the default and routed as a cluster-gated opt-inI'll restructure around the two modes you describe, and I'm keeping your framing of routed almost verbatim: routes annotation (inherited onto pods at Create by the kubeovn webhook) provides the return path, so no SNAT and the client source IP is preserved; the "missing SNAT → black-hole" finding is NAT-mode-only; whole-subnet reachability, ICMP, and no per-target enumeration are real routed advantages; port_security-off on the gateway port is acceptable for v1 with containment from Cilium's sender-side egress enforcement; scoped port_security (declared remote CIDRs added to the port's allowed addresses) is the target state, with AAP as the right mechanism once kube-ovn grows CIDR support. And routed is transparent L3 for TCP/UDP/ICMP/SCTP only — ESP/GRE/OSPF/VRRP still won't cross the fabric (same conntrack finding). The one change I'd propose to your ordering: NAT is the default; routed is opt-in and gated by a cluster-level switch so a platform admin can disallow the port-security-relaxing, kube-ovn-coupled mode entirely. Reasoning:
Two supporting points, both of which you already raised:
Net: your two-mode structure stands unchanged — I'm only flipping which is the default and adding a cluster gate for routed. I'll be honest in the doc that NAT's cost is losing the original source IP (breaks source-IP ACLs / One NAT boundary worth stating up front, since it's a whole app class rather than a nit — protocols that advertise their own topology. Kafka ( 2. Day-2 reconfiguration: agreed — VyOS HTTP API driven by the platform controllerAdding a Day-2 section built on the mechanism you settled on: the platform controller renders target config from values and pushes it over the VyOS HTTPS API; the API key is delivered once at first boot via cloud-init; the API is reachable by the platform only (boundary stated in Security). v1 may fall back to "config change = VM restart" if called out explicitly. A few points I want to bake in from validation/research so the section is precise:
3. Low-level entities & controller: agreedAdding the section with the shape you described: input stays in the catalog-app values (no new CRD — the aggregated API server just mirrors HelmReleases); validation = values.schema plus controller-side reconcile checks surfaced as status conditions; the controller validates declared remote CIDRs (disjoint from pod/service/join/node; cross-tenant overlap is fine since routes are namespace-scoped), sets the namespace routes annotation, scopes/relaxes port_security on the gateway VM port, and cleans everything up on app deletion; routes apply only to newly created pods, so no auto-restart — the controller reports pods whose annotation lags the namespace ("pods pending route") and the user rolls them; CNI-touching operations sit behind a small backend interface so a future CNI can implement them natively. One clarification that ties back to §1: the controller's CNI-mediation duties (routes annotation, port_security, CIDR validation, pods-pending-route) exist specifically to serve routed mode. In a NAT-only cluster the controller reduces to the thin day-2 driver (render values → VyOS API, manage the tunnel's external exposure). That's a further reason to gate routed at the cluster level — clusters that don't enable it never take on the CNI-mediation surface. Nits
I'll push a revision restructuring the doc along these lines. Happy to keep discussing the default (§1) if you feel strongly the other way, but that's the reasoning. |
|
Operator data point: we run the routed (no-NAT) mode in production today, and the proxied/NAT method would be a regression for us. Our current setup is a tunnel-terminating VM used as a plain gateway: On the "is the proxied (DNAT+SNAT) method suitable" question: for our use cases, no. Losing the original source IP and dropping to enumerated So we'd strongly favor routed being a first-class Phase-1 mode, not a fallback. On the NAT-default-vs-routed-default question, a cluster-level gate for routed is fine from our side, but routed shouldn't be relegated to "fallback only." |
…+ NAT) Split the tenant site-to-site connectivity proposal into two co-equal catalog apps over a shared foundation — site-router (routed, source-IP preserving, kube-ovn) and site-gateway (NAT, CNI-agnostic) — instead of a single NAT gateway with routed as a fallback, and rename the proposal to tenant-site-connectivity. Adds day-2 configuration (VyOS HTTPS API driven by a controller), a low-level-entities/controller section, and publishes the tunnel endpoint via the structured external-exposure primitives (#29) instead of a bespoke LoadBalancer. Tightens secret handling: the chart templates the Secret from values (no tenant-created Secret), the WireGuard key is autogenerated in-chart, and the IPsec PSK is the only genuinely tenant-supplied secret. Reorders rollout to lead with routed + MSS clamping + tunnel observability. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
Pushed a restructure ( 1. Modes → two co-equal apps, and routed leads Phase 1.
2. Day-2 reconfiguration — new section. A platform controller drives the guest's VyOS HTTPS API (atomic set/delete batch; cloud-init is first-boot only; the API key is generated at runtime and seeded once; the API is reachable by the platform only). 3. Low-level entities & controller — new section. Input stays in the catalog-app values (no new CRD); validation = values schema + controller reconcile checks surfaced as status conditions. The controller sits behind a small backend interface so the neutral VyOS-driving core is shared; Nits:
PTAL — happy to iterate. |
myasnikovdaniil
changed the title
|
Thanks for the restructure. Splitting into two co-equal apps with routed as Phase 1 matches what we run in production, so a few operator notes. Routed dataplane validation. The doc notes the e2e prototype so far exercised the NAT dataplane, while Phase 1 now leads with Scoped port_security. +1 on making scoped CIDR disjointness. For the overlap-with-cluster-networks footgun, restricting the selectable remote CIDRs at the values/UI layer on top of the controller-side disjointness validation lets the two checks reinforce each other, with the controller validation as the authoritative one. |
I agree, I will fold this clarification on next round of review, thanks |
- Status Draft -> Review - Declare #29 (with UDP ExposureClass) an explicit Phase 1 blocker, with a stopgap UDP LoadBalancer fallback if it slips - Reframe routed (site-router) as validated in a production implementation; make scoped port_security a Phase 1 acceptance criterion, not a Phase 4 relaxation - Add an Image lifecycle subsection: VyOS image built in-repo (Talos-image pattern), CVE patching via re-pin+rebuild, update = new disk => VM restart - Specify tenant-immutable enforcement for the guest management API (no tenant-namespace NetworkPolicy; API key in a platform-only Secret), to be re-verified against the production implementation - Commit to singular peer: (one tunnel per instance); add a multi-peer open question instead of retrofitting peers:[] later - Enrich the remote-CIDR deny-set (link-local/metadata, LB pool, loopback, default route) and surface rejection as a status condition - Keep IPsec the default backend; WireGuard is a first-class alternative in Phase 3, never promoted to default Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
Updated design with few improvements |
|
This is a strong revision — all three structural points landed in the doc itself, and several additions go beyond the review (the remote-CIDR deny-set, the platform-only management-API enforcement via a cluster-scoped policy, the topology-advertising app class, #29 as an explicit UDP-gated Phase-1 blocker). Two adjustments before I approve: 1. Ship the two apps in sequence, not both in one design pass. The two-app split over a shared foundation is the right call — keep it. But land only 2. Don't make scoped Two nits still only in the thread, not the doc — fold both in on the next pass: the portal-side image/cloud-init sync, and NETMAP-style 1:1 subnet mapping as NAT future work. Otherwise this is ready — thanks for the thorough turnaround. |
1 similar comment
|
This is a strong revision — all three structural points landed in the doc itself, and several additions go beyond the review (the remote-CIDR deny-set, the platform-only management-API enforcement via a cluster-scoped policy, the topology-advertising app class, #29 as an explicit UDP-gated Phase-1 blocker). Two adjustments before I approve: 1. Ship the two apps in sequence, not both in one design pass. The two-app split over a shared foundation is the right call — keep it. But land only 2. Don't make scoped Two nits still only in the thread, not the doc — fold both in on the next pass: the portal-side image/cloud-init sync, and NETMAP-style 1:1 subnet mapping as NAT future work. Otherwise this is ready — thanks for the thorough turnaround. |
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
Thanks — agreed on point 1 and both remaining nits. I’ll make a small follow-up that:
On point 2, I rechecked the kube-ovn v1.15.10 path and agree with your mechanical finding. The latest revision ( I added one important qualification to the containment claim. With an empty OVN Cilium still provides valuable sender-side destination containment: the gateway’s endpoint identity is compiled into its eBPF program, so spoofing the packet source should not grant direct access to unrelated tenant endpoints, nodes, or the Kubernetes API. But it is not complete containment—the gateway policy permits The proposal now makes the compensating controls explicit Phase-1 requirements:
So full relaxation is accepted for v1, but the trust boundary is explicit: without the guest source filter, a compromised remote tunnel peer can inject arbitrary-source traffic up to the destinations allowed by the gateway’s remaining policies. Scoped kube-ovn port security remains the immediate hardening follow-up once CIDR AAP support exists. |
|
Thanks — confirming the state after Point 2 ( Point 1 + both nits — still pending in the doc. Per your note above these land in the follow-up; just recording the gate so we don't lose it:
Holding approval until that follow-up is pushed; everything else is ready. |
Timofei Larkin (lllamnyp)
left a comment
Timofei Larkin (lllamnyp)
left a comment
There was a problem hiding this comment.
Requesting changes — the Phase-1 dependency targets an API group that's being removed.
Flagging this on an already-approved PR because the dependency moved underneath it rather than because of anything in the design itself. The good news is that this makes Phase 1 less blocked, not more.
The issue. The proposal declares #29's ServiceExposure / ExposureClass an explicit Phase-1 blocker and builds the tunnel entry point on them. cozystack/cozystack#3218 (open, approved) removes the network.cozystack.io API group — ExposureClass + ServiceExposure (#3081) — along with its reconciler and all five backends, in favor of native Service type: LoadBalancer + loadBalancerClass with an admin-provisioned pool. Rationale and discussion in cozystack/cozystack#3164. As written, this proposal's Phase 1 gates on primitives that will not exist.
Why this is good news. The blocker was "the exposure class must support UDP (IKE/NAT-T 4500, WireGuard), which #29 does not yet provide." A native Service carries protocol per port, and the LB implementations already in use handle UDP. So the dependency doesn't need re-pointing at a replacement contract — it dissolves. Phase 1 becomes unblocked.
That inverts the current fallback paragraph specifically. It reads:
If #29 stalls, the fallback is a bespoke UDP
LoadBalancerService on the gateway as a stopgap — explicitly a temporary path to be migrated onto the #29 contract once available, not a parallel long-term mechanism.
Post-3218 the "temporary stopgap" is the sanctioned mechanism, and the contract it was to migrate onto is being deleted. This needs inverting rather than trimming.
Places to touch (line numbers against the current head):
- L23 — Scope: drop the blocking-dependency framing; state that the tunnel endpoint is a
Service type: LoadBalancer+loadBalancerClasson the gateway, and that Phase 1 carries no exposure blocker. - L58 — "the #29 exposure of the tunnel endpoint" in the shared-foundation list.
- L80 — the mermaid node
#29 tunnel exposure (ServiceExposure, UDP). - L92 — "Tunnel entry point. Published via #29
ServiceExposure+ExposureClass(UDP), not a bespokeLoadBalancer" — this is now exactly backwards. - L112 — "the gateway's #29-exposed UDP endpoint."
- L230 — Rollout Phase 1's "(blocked on #29 shipping UDP exposure)".
- L240 — the open question "Tunnel exposure via #29 (blocking)". The LB-pool/quota interaction it raises is still a real open detail; it just attaches to the pool/
loadBalancerClassnow. - L121 / L241 — the topology-advertising discussion leans on "#29's
targetis per-named-listener, not per-broker/replica" and floats a tunnel-backedExposureClassas future unification. The conclusion (routed is the transparent path for this class) is unaffected and correct, but the reasoning needs restating without theExposureClassvocabulary.
A larger question this opens, worth at least a line in the doc: does the gateway need an inbound listener at all?
A site-to-site tunnel is symmetric once established — inbound "external site → tenant app" works identically regardless of which side dialled. If the gateway is the initiator, there's no listener, no Service, no pool, and no exposure story of any kind.
This design is already most of the way there: forced UDP encapsulation is mandatory unconditionally because of the native-ESP-dropped-by-the-overlay finding, and forced UDP encap is NAT-traversal mode. A NAT'd initiator needs the machinery this proposal already requires for an unrelated reason. WireGuard is more natural still — the responder identifies a peer by key and learns the endpoint from the handshake. It would also simplify the HA story: no exposure address to reconverge, the standby just dials out.
The cost is real and worth stating rather than glossing: the initiator leaves via the cluster egress masquerade wearing the node's address, shared and unstable, and remote sites routinely pin the peer's source IP in their IKE config and firewall ACL. So dial-out trades an inbound exposure dependency for a stable, tenant-owned egress identity — which this proposal currently defers to Phase 4 as a non-goal. It's also the one thing a Service type: LoadBalancer structurally cannot express: an LB Service holds an ingress address, and there's no Service to hang an egress identity on. That's the gap #35 is proposing an object for.
Not asking to resolve that here. But the choice of who initiates determines whether Phase 1 has any platform dependency at all, so it deserves to be a stated decision rather than an implicit assumption that the remote side always dials in.
Happy to review the follow-up — this fits naturally alongside the three items already recorded above (the Phase-2 reframing, portal-side image/cloud-init sync, and NETMAP-style 1:1 subnet mapping).
4 participants
Summary
Adds a design proposal for tenant-managed external network connectivity:
terminating a site-to-site VPN (IPsec or WireGuard) inside a KubeVirt VM that
NAT-bridges an external site to the tenant's managed apps, in both directions,
for any L4 protocol — without granting tenants privileged host-cluster pods.
The proposal covers:
terminated in a VM guest (contained privilege)
packages/apps/site-gatewaycatalog app with a values-schema sketchfills the tenant-namespace-apps + NAT-egress space design-proposal: cross-cluster mesh for tenant access to host services #7 defers
The design was prototyped and validated end-to-end (two gateway VMs + a real
managed Postgres); results are included, notably the finding that native ESP is
dropped by the Cilium/kube-ovn overlay (forced UDP encapsulation required for
IPsec; WireGuard sidesteps it).
Test plan
Design proposal; no code. Implementation testing (helm-unittest across backends,
e2e of the two-VM topology asserting inbound/outbound + SNAT-required + MSS
behavior) is scoped in the proposal and will follow in implementation PRs.
🤖 Generated with Claude Code
Summary by CodeRabbit