Repository navigation
Harden load balancers served from project namespaces #1107
Description
Gateways can now serve load balancers that live in a project's namespace (#1106). Review turned up hardening that should land before projects can write those load balancers directly. None of it is reachable until the platform starts writing into project namespaces.
Part of network load balancers.
Work
- Address squatting. A project can claim any address no one else serves yet, and the gateways will advertise it. Only accept addresses from the platform's allocated load balancer pools.
- Trusted gateway namespaces. A namespace is treated as trusted only because it holds a gateway. Anyone able to create a gateway in a project namespace would skip every check. Trust only the configured gateway namespace, or test that projects can't create gateways.
- Missing permission fails quietly. If the gateway's role lacks read access to networks, it logs and then stops serving every load balancer. Fail the startup check instead.
- Project deletion can hang. If no gateway node remains, a deleting project load balancer keeps its finalizer and the project namespace stays in Terminating.
- Optional rollout switch. A flag to turn project-namespace serving on per location, so staging can use it before stable sites do.
- Tests for a project or load balancer deleted mid-flight, a network deleted after acceptance, and a gateway appearing in a project namespace.
- Scale. Each load balancer change lists every load balancer in every project to find address conflicts. Index by address if it gets slow.
Activity
privateip commentedon Oct 28, 2026
Verdict: valid, checked against
mainat 389a297. Each item holds in the code #1106 added:tenantVIPConflicts(internal/controller/servicescope.go) only refuses an address another service already serves. Nothing checks a project's VIP against an allocated range, so the first claim on any unused address is accepted and advertised.listServedNetworkServicesandservingGatewaystreat every namespace that holds aNetworkGatewayas a gateway namespace. ANetworkGatewaycreated in a project namespace makes that namespace's services trusted, so the VPC ownership and VIP checks inacceptedCondition(internal/controller/networkservice_controller.go) are skipped.checkWatchPermissions(cmd/galactic-gateway/main.go) only logs a denied watch. Withoutvpcsaccess the first project service starts a VPC informer that never syncs, which blocks every reconciler.NetworkServiceReconciler.Reconcilereturns beforereconcileDeletewhennodeServingGatewayfinds no gateway for this node. With noNetworkGatewayleft, nothing removesgalactic.datum.net/networkservice-teardown, so the namespace stays in Terminating.servicescope_test.gohas no cases for deletion mid-flight, a VPC deleted after acceptance, or a gateway in a project namespace.acceptedConditionlists everyNetworkServicein the cluster for each project service it reconciles. This is a cost note, not a defect.These were confirmed by reading the code. A fix plan follows.
privateip commentedon Oct 28, 2026
Fix plan:
The gateway gets three settings:
GALACTIC_GATEWAY_PROJECT_SERVICESturns project-namespace serving on. It defaults tofalse.GALACTIC_GATEWAY_NAMESPACEnames the one trusted gateway namespace. The DaemonSet sets it from the pod's namespace.GALACTIC_GATEWAY_PROJECT_VIP_PREFIXESlists the allocated load balancer prefixes. It is required when the switch is on.How each item is handled:
Accepted=Falsewith reasonVIPOutsidePool.NetworkGatewayin any other namespace is ignored. With the switch off, a gateway serves only services in its own namespace, as it did before Serve load balancers from project namespaces #1106.vpcscheck runs only when the switch is on.GALACTIC_GATEWAY_PROJECT_SERVICESabove.Rollout: before the platform writes load balancers into project namespaces, the staging infra overlay must turn the switch on and set the prefixes.
privateip commentedon Oct 28, 2026
One addition to the plan: turning
GALACTIC_GATEWAY_PROJECT_SERVICESoff after it was on left project services' advertisements in place, because the orphan sweep keeps any advertisement whose service still exists. With the switch off, the sweep now withdraws every project service's advertisements.TestTenantService_SwitchedOffWithdrawsRoutescovers this.