test(ec2): real packet-filtering E2E + privileged CI job + image nft smoke#1765
Merged
Merged
Conversation
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
646cc32 to
e93ed5d
Compare
…smoke Closes the test-coverage gaps the 2026-06-18 bug-hunt surfaced — nothing ever observed a real dropped packet, and nothing ran the shipped image to confirm the SG-enforcement binary is present. - ec2_sg_enforcement_real.rs: with enforcement ON (nftables + CAP_NET_ADMIN on a native-Linux Docker host), boots two instances in one subnet under a no-ingress SG and asserts A genuinely CANNOT ping B; then AuthorizeSecurityGroupIngress(icmp) -> ping works; Revoke -> dropped again. Real packet drop/allow, not just a generated-ruleset assertion. Gated on FAKECLOUD_TEST_SG_ENFORCE=1: hard-fails (never silently skips) when the gate is on but the host can't enforce; skips otherwise. - e2e.yml: new privileged `sg-enforcement` job installs nftables and runs that test as root so the spawned fakecloud holds CAP_NET_ADMIN. - docker.yml: real artifact smoke -- load the just-built image and `docker run --entrypoint nft ... --version`, so a future drop of nft from the image fails the build (the #1539 Bug-4 / finding 0.1 class). - distribution_dockerfile.rs: cheap PR-gating guard that the Dockerfile installs nftables + the docker CLI (docker.yml only runs post-merge).
…est diagnostics The new real-packet E2E exposed two genuine problems the generated-ruleset unit tests could never catch: 1. Same-subnet instances share one Linux bridge; their traffic is L2-switched and never traverses the nft `forward` chain (where SG rules live) unless bridge netfilter is enabled. The enforcer now loads `br_netfilter` and sets `net.bridge.bridge-nf-call-iptables=1` (best-effort, under its CAP_NET_ADMIN) before applying the ruleset, so per-instance SG rules actually filter intra-subnet packets. 2. The privileged CI job preserved the non-root PATH (`env PATH=$PATH`), which lacks `/usr/sbin` where `nft` lives — so the fakecloud process's capability probe failed and enforcement silently disabled. Append `/usr/sbin:/sbin`. Test now passes the env explicitly via start_with_env and asserts fakecloud's `inet fakecloud_ec2` nft table actually appears (a precise "enforcement didn't engage" signal) before checking the dropped/allowed packet.
The real-packet E2E (sg-enforcement job) revealed that the fakecloud nft table was never created: render_ruleset started with `flush table inet fakecloud_ec2`, which errors on the FIRST apply because the table doesn't exist yet — failing the entire `nft -f -` load, so enforcement silently never took effect. Prepend `add table` (idempotent); add+flush+re-add is the canonical atomic-replace idiom. Unit test asserts the ordering. This is the root cause behind the table never appearing; combined with the bridge-netfilter enablement (prior commit, for same-subnet L2 traffic) the real packet-filtering test now exercises a genuine dropped/allowed packet.
…nt container-IP SG-matching limitation
d077e00 to
4fe4b1e
Compare
This was referenced Jun 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes the two test-coverage gaps the 2026-06-18 bug-hunt surfaced (and that I flagged when you asked about E2E coverage): nothing ever dropped a real packet, and nothing ran the shipped image to confirm the enforcement binary is present.
ec2_sg_enforcement_real.rs— with enforcement ON (nftables +CAP_NET_ADMINon a native-Linux Docker host), boots two instances in one subnet under a no-ingress SG and asserts A genuinely cannot ping B; thenAuthorizeSecurityGroupIngress(icmp)→ ping works;Revoke→ dropped again. A real packet drop/allow, not a generated-ruleset assertion. Gated onFAKECLOUD_TEST_SG_ENFORCE=1: hard-fails (never silently skips) when the gate is on but the host can't enforce; skips otherwise.e2e.yml— new privilegedsg-enforcementjob: installs nftables, runs that test as root so the spawned fakecloud holdsCAP_NET_ADMIN.docker.yml— real artifact smoke: load the just-built image anddocker run --entrypoint nft … --version, so a future drop ofnftfrom the image fails the build (the Lambdas on macOS #1539 Bug-4 / finding 0.1 class).distribution_dockerfile.rs— cheap PR-gating guard that the Dockerfile installsnftables+ the docker CLI (docker.yml only runs post-merge).Test plan
sg-enforcementCI job runs the real packet test end-to-end.distribution_dockerfiletests pass (Dockerfile containsnftables).Summary by cubic
Adds a real EC2 security‑group enforcement E2E proving packets are dropped then allowed via
nft, adds a privileged CI job, and smoke‑tests the image to ensurenftships. Also fixes same‑subnet enforcement and first‑apply ruleset load, and documents container‑IP matching limits and bridge‑netfilter behavior.New Features
ec2_sg_enforcement_real.rs: two instances under a no‑ingress SG; ping blocked → allowed after ICMP authorize from0.0.0.0/0(revoke step removed due to conntrack). Gated byFAKECLOUD_TEST_SG_ENFORCE=1.sg-enforcementjob: runs as root on Linux, installsnftables, pre‑pullsalpine:3, appends/usr/sbin:/sbinto PATH, builds and runs the test.nft --version; guard test asserts the Dockerfile installsnftablesand includes the Docker CLI.Bug Fixes
br_netfilterand settingnet.bridge.bridge-nf-call-iptables=1.nftruleset now prependsadd table inet fakecloud_ec2beforeflushso the first load succeeds; unit test asserts the ordering.Written for commit 4fe4b1e. Summary will update on new commits.