feat(networking): Windows L2Bridge egress with router-safe VFP ACLs (opt-in)

[luthermonson]

Problem

Windows container egress cannot be software-filtered on the current HNS NAT
network. VFP (the Virtual Filter Platform) does not engage on a NAT network, so
the per-endpoint Switch ACLs ephemerd already applies are inert — enforcement
there rests entirely on the Hyper-V firewall. We need an enforcement point where
software ACLs actually bite.

Fix (opt-in, NAT stays default)

Add an L2Bridge networking path, gated behind a per-pool flag
(network.l2bridge_egress). On L2Bridge the container gets a VFP-managed
vSwitch port, so per-endpoint ACLs enforce. This was proven on metal
(node mfl-win-amd64-1, Server 2025): with the correct ACL model, a
Hyper-V-isolated container reached 1.1.1.1 / public HTTPS while
container→Grafana/Incus/Proxmox (RFC1918) was blocked.

Merging this flips nothing — NAT remains the default and the NAT init/setup/
ACL path is untouched. A pool only reaches L2Bridge by setting
l2bridge_egress = true.

The router-safe ACL model (the load-bearing change)

VFP is default-DENY the moment any ACL is present — a pure-Block set (today's
buildEgressBlockPolicies) would black-hole the internet too. buildL2BridgeEgressACLPolicies
(a pure, unit-tested function) emits this precedence ladder as RuleType=Switch
VFP ACLs (lower Priority = higher precedence, first match wins):

Priority	Action	Dir	Remote / Ports	Proto	Purpose
90	Allow	Out	RemotePorts 67,68	UDP (17)	DHCP lease/renew survives the block
90	Allow	In	LocalPorts 67,68	UDP (17)	DHCP reply
95	Allow	Out	extra-allowed CIDR	256	operator carve-outs (future use; default none)
100	Block	Out	10/8, 172.16/12, 192.168/16, 169.254/16	256	RFC1918 + link-local, whole supernets
65500	Allow	Out	0.0.0.0/0	256	internet
65500	Allow	In	0.0.0.0/0	256	TCP return traffic (SYN-ACK)

This matches the Linux end-state (firewall_linux.go): block all of RFC1918
including the LAN router, permit only the internet.

• Whole supernets, no carve-out. No gateway/own-subnet exclusion (unlike NAT).
  On L2Bridge the container is a LAN peer, so carving out the subnet would expose
  the same-subnet management plane and the router. This is the fix for the
  same-subnet management-plane bug.
• Both allow-any rules are mandatory. Without the low-precedence pair the port
  default-denies everything; the inbound allow is the non-obvious one — drop it
  and every outbound TCP connect fails when the SYN-ACK returns.
• DNS → public resolvers. The endpoint's Dns.ServerList is set to a
  configurable public resolver (default 1.1.1.1, 8.8.8.8), so DNS rides the
  0.0.0.0/0 allow and the container never needs the LAN router. No allow to
  the gateway/router.
• No container-to-container allow (unlike Linux) — on L2Bridge that is LAN
  access, which we block.

Fail-closed, no post-lease window

The ladder is static — it does not depend on the leased gateway or DNS (DNS is
public, the router gets no allow) — so the ACLs are applied at endpoint creation,
before the container starts. Any ApplyPolicy error tears down the endpoint and
refuses the job.

Network shape

L2Bridge network is hcn.L2Bridge bound to the host NIC via a hcn.NetAdapterName
network policy (recipe §1). IPAM = DHCP: the network declares no static subnet/
routes and the endpoint carries no IpConfigurations, so the container leases its
IP, gateway, and default route from the LAN DHCP server. Network name is
ephemerd-l2bridge (distinct from the NAT ephemerd).

Config surface (what mayfly must render)

New [network] keys (per-pool → rendered into the per-VM ephemerd config.toml,
same flow as the existing [vm.linux] toggle and windows_runner sizing):

• l2bridge_egress (bool, default false) — the opt-in.
• host_nic (string) — required when l2bridge_egress = true; the host adapter
  name to bridge onto. No default (host-specific; do not hardcode "Ethernet 2",
  the spike's hot-added test NIC). Missing → init fails closed.
• public_dns ([]string, default ["1.1.1.1","8.8.8.8"]).
• extra_allowed_destinations ([]string, default []) — future use.

Wired through networking.Config (L2BridgeEgress, HostNIC, PublicDNS,
ExtraAllowedCIDRs) at both networking.New call sites in cmd/ephemerd/main.go.
Documented in config.example.toml. mayfly rendering is a follow-up (not in
this PR): add the four keys to the per-pool Windows config schema and render them
into [network]; host_nic should be a required field when the pool opts in.

Honest caveats

• DHCP-on-L2Bridge was NOT validated on metal. The spike proved L2Bridge + VFP
  enforcement with a static IP. DHCP is the chosen v1 IPAM and will be validated
  at the gated cutover. If DHCP-on-L2Bridge misbehaves there, static-pool IPAM is
  the proven fallback (recipe §6 recommends static-first).
• The 3-tier carve-out (allow-/32 above block) was not run end-to-end on metal —
  the 2-tier (high-prio blocks over low-prio allow-any) was. The default posture here
  has no carve-outs (empty extra_allowed_destinations), so the default set is the
  proven shape plus the DHCP allow.
• Anti-spoof: documented gap. hcsshim v0.14.0-rc.1's hcn does not expose a
  ready source-IP/MAC anti-spoof endpoint/port policy in a form I could apply without
  guessing the schema, so it is not implemented here rather than faked. On L2Bridge
  the container is a routable LAN peer that could source-spoof within the L2 domain;
  egress enforcement rests on the VFP ACLs. Follow-up: evaluate an HNS L2Bridge
  MAC/IP anti-spoof port policy.

Deviations from the recipe

• Did not repurpose buildEgressBlockPolicies for NAT. Rewriting it into the
  ladder (or applying the ladder on NAT) would blackhole the NAT gateway 10.88.0.1
  (inside the whole-10/8 block, no carve-out) if VFP ever engaged. Kept the NAT
  block-only builder and path exactly as-is; the ladder is a new function on
  the L2Bridge path only. This better satisfies "NAT stays intact".
• DHCP shape (no static IPAM, no endpoint IP) instead of the recipe's static
  IpConfigurations — per the user's DHCP v1 choice; see caveat above.

Tests

pkg/networking/network_windows_test.go (pure, no HCN calls):

• TestL2BridgeEgressACLPolicies_LadderShape — both allow-any (Out+In), both DHCP
  allows, all four supernets blocked; exact protocols/priorities/directions.
• TestL2BridgeEgressACLPolicies_NoGatewayOrSubnetCarveOut — no allow to any RFC1918
  address, blocks are plain CIDRs (no exclusion ranges leaked in).
• TestL2BridgeEgressACLPolicies_Precedence — DHCP/extra < block < allow-any.
• TestL2BridgeEgressACLPolicies_ExtraAllowed — carve-outs emitted Out, above block.
• Existing TestBuildEgressBlockPolicies (NAT) unchanged and passing.

go build ./pkg/networking/... ./pkg/config/..., go vet ./pkg/networking/..., and
go test ./pkg/networking/... all clean (native GOOS=windows). (go build ./...
hits a pre-existing missing-embed-artifact error in pkg/vm, unrelated to this change.)

Not in this PR

No live node touched. The live cutover is a separate, gated step.

[luthermonson]

On-metal validation — mfl-win-amd64-101 (Proxmox VM 101, Server 2025 26100)

Ran this branch on the real node with a second vNIC hot-added on the LAN bridge (net1, enumerated as Ethernet 2, removed again afterwards). Containers were Hyper-V-isolated, on the already-cached ephpm/ephemerd:runner-ci-windows, created through networking.New() → Manager.Setup() so the endpoint received the generated ACLs — nothing hand-applied on the treatment run. The production daemon, its NAT network ephemerd, and the nightly were left alone.

Two blocking findings

1. DHCP IPAM cannot create the network at all. initL2Bridge declares no Ipams; HNS rejects that outright:

creating HCN L2Bridge network on "Ethernet 2": hcnCreateNetwork failed in Win32:
The network does not have a subnet for this endpoint. (0x803b0005)
{"Success":false,"ErrorCode":2151350277}

No network, no endpoint, no container. This is the first thing that happens when a pool sets l2bridge_egress = true, so as merged the feature cannot start a job. Give the network a subnet + default route and everything downstream works — HNS then assigns endpoint addresses itself with no IpConfigurations (it handed out 192.168.0.19, 192.168.5.114, 192.168.14.56, 192.168.15.65 across runs). I did not switch IPAM over: it means handing containers real LAN addresses, which needs a range the LAN DHCP server won't also hand out. That's your call, so it's recorded as UNRESOLVED in the code.

2. The two DHCP ACLs blackhole the port. With a subnet in place so the rest of the path could run, the generated ladder blocked everything, internet included. Bisected on the same endpoint and code path, only these two rules varying:

ACL set	Grafana :3000	Incus :8443	Proxmox .1/.2:8006	Router 192.168.1.1:80/443	ephemerd host :135	1.1.1.1:443	8.8.8.8:53	DNS by name	public HTTPS
none (baseline control)	OPEN	OPEN	OPEN	OPEN	OPEN	OPEN	OPEN	OK	HTTP 200
allow-any only	OPEN	OPEN	OPEN	OPEN	OPEN	OPEN	OPEN	OK	HTTP 200
ladder as generated	blocked	blocked	blocked	blocked	blocked	blocked	blocked	FAIL	FAIL
ladder + explicit 192.168.1.1/32 allow @95	blocked	blocked	blocked	blocked	blocked	blocked	blocked	FAIL	FAIL
ladder minus the DHCP rules	blocked	blocked	blocked	blocked	blocked	OPEN	OPEN	OK	HTTP 200
original by-hand 7-policy set	blocked	blocked	blocked	OPEN	OPEN	OPEN	OPEN	OK	HTTP 200

The "allow-any only" row shows Allow rules work and the port is not default-denied just by having ACLs. The by-hand row shows the enforcement mechanism is intact. The only variable that turns a working ladder into a total blackout is the pair of port-scoped UDP 67/68 allows — and note that with them present even an explicit higher-precedence Allow for the gateway does not survive.

HNS accepts them (ApplyPolicy returns success); the VFP rule set it produces drops everything. It fails closed rather than open, so it breaks jobs instead of leaking — but it makes the feature unusable.

Nothing here needs them: the endpoint is addressed by HNS IPAM, not by a DHCP client in the container. Removed in 138a737, with a regression test asserting every rule in the ladder is address-scoped and none is port-scoped.

The good news

Once those two rules are gone, the router-safe model does exactly what it claims — and this is the part that had never been validated. Blocking the whole of 192.168.0.0/16 including the container's own subnet and its default gateway does not break egress: the container still routes through 192.168.1.1 as a next hop while being unable to address it. Grafana, Incus, both Proxmox hosts, the LAN router and the ephemerd host itself are all unreachable; 1.1.1.1, 8.8.8.8, DNS-by-name via the public resolvers, and public HTTPS all work. That's a strictly better posture than the by-hand proof, which left the router and host reachable.

Mechanistic tell

vfpctrl /list-vmswitch-port enumerates the container's port on the L2Bridge switch (it errors on NAT):

Port name            : E37525A9-E8DA-428A-83EF-C8F6111C07AB
Switch Friendly name : ephemerd-l2bridge
Port type            : Synthetic
MAC address          : 00-15-5D-6D-BE-D7
VM name              : l2base-ctr@vm
Command list-vmswitch-port succeeded!

Also worth a look

setup() claimed installFirewallRules is a no-op. It isn't — firewall_windows.go programs Hyper-V firewall rules — but hyperVEgressRules is called with DefaultSubnet/defaultGateway, so on L2Bridge, where the container holds a LAN address, those rules match nothing. The per-endpoint ACLs really are the only thing enforcing. Comment corrected; the NAT-subnet scoping is left as-is since it's the NAT path's concern.

Not merging — #1 still needs an IPAM decision from you.