Catch silent edge misconfiguration before it reaches customers

[scotwells]

The problem

The edge applies real protections on customers' behalf — firewall rules, connector routing, branded error pages, per-listener certificate isolation. But our confidence that those protections are actually in force has rested almost entirely on "the request worked." A request can succeed while the protection behind it never applied, applied only partly, or landed in the wrong place. That whole class of silent failure has been invisible to us, and our edge tests couldn't catch it because they ran against a simplified setup that didn't resemble how the edge really runs.

Two gaps, reinforcing each other:

• No way to ask the edge what it believes it should be running. We could observe traffic, but not the intended configuration behind it — so we couldn't tell "healthy response" apart from "healthy response with the protection silently missing."
• No faithful place to prove edge behavior. Tests ran on a single-stack stand-in, not the real two-cluster topology, real gateway version, real firewall data plane, or real config-delivery path. Every gap between test and production was a place a bug could hide.

The work

A stack of five PRs that closes both gaps and then proves the edge's core guarantees with real traffic:

• Let the edge report the protections it was told to run #224 — Report the protections the edge was told to run. A read-only endpoint that exposes the edge's intended configuration set. The reference point everything else builds on.
• Confirm the edge is actually running the configuration it was given #225 — Confirm the edge is running what it was given. Compares intended vs. actually-serving config and names any gap in plain terms (missing, wrong count, wrong place, rejected, dropped) instead of leaving a surprising result a guess.
• Stand up a test environment that mirrors how the edge really runs #226 — A test environment that mirrors how the edge really runs. Two clusters wired like production, the production gateway version, the real firewall data plane, and the real config-delivery path.
• Carry edge policy from the control plane out to the edges #227 — Carry edge policy from the control plane out to the edges. The delivery wiring so tests see the same policy-propagation behavior customers depend on.
• Prove the edge's core guarantees with real traffic #228 — Prove the edge's core guarantees with real traffic. Four end-to-end tests: the firewall enforces, an offline origin fails cleanly, the branded error page shows, and one bad certificate can't break its neighbors — each judged from a real request, backed by proof the edge was configured the way the test assumed.

Outcome

Silent edge misconfiguration gets a name and a place to surface — in testing, before it reaches customers — instead of hiding behind a green response in production.

Known follow-up

The new prod-fidelity environment and its real-traffic scenarios are not yet wired into CI. CI still runs the older single-stack make test-e2e; the two-cluster env and test/e2e-edge/ chainsaw scenarios run locally via task test-infra:e2e. Wiring this stack into CI so every PR exercises the real edge path is the natural next step once the stack lands.

[ecv]

Status update — the five-PR stack has landed:

• Let the edge report the protections it was told to run #224 report intended config — merged
• Confirm the edge is actually running the configuration it was given #225 config parity check — merged
• Stand up a test environment that mirrors how the edge really runs #226 prod-fidelity two-cluster env — merged
• Prove the edge's core guarantees with real traffic #228 real-traffic edge guarantees — merged
• Carry edge policy from the control plane out to the edges #227 carry edge policy control-plane→edge — open (active)

Plus #276 lifted the e2e suite onto the env and wired it into CI (test-e2e.yml), retiring the #268 bridge. #275 (lift suite onto env) is closed complete.

Both gaps in the original problem are now closed in testing: the edge can report its intended config (#224) and the parity gate names any drift (#225), proven with real traffic on the prod-fidelity env (#228).

Remaining follow-up: #227 (policy delivery) and #248 (federated Karmada CI job) are still open. Keeping this umbrella open until those land.

[ecv]

Update — #227 (carry edge policy control-plane→edge) merged (89c1c0a), landing the Karmada delivery wiring (config/federation/ — ClusterPropagationPolicy + resource interpreters) on trunk. E2E Tests green on the new HEAD.

Full stack status:

• Let the edge report the protections it was told to run #224 report intended config — merged
• Confirm the edge is actually running the configuration it was given #225 config parity check — merged
• Stand up a test environment that mirrors how the edge really runs #226 prod-fidelity env — merged
• Carry edge policy from the control plane out to the edges #227 policy delivery — merged
• Prove the edge's core guarantees with real traffic #228 real-traffic edge guarantees — merged
• test(infra): prod-fidelity edge test env + e2e suite #276 lift suite onto env + CI — merged; fix: install downstream WAF data plane in e2e CI (#263) #268 bridge retired

Only remaining child: #248 (federated e2e CI job). With #224–#228 + #227 all on main, #248 collapses to essentially one workflow (test-e2e-federated.yml) plus its Taskfile federation tasks — everything else it carried is now upstream. Closing this umbrella once #248 lands.