Edge serves stale TrafficProtectionPolicy config during upstream→edge propagation with no lag visibility

[ecv]

Summary

TrafficProtectionPolicy spec changes made on the upstream control plane take an unbounded, unobservable amount of time to reach the edge data plane. During that window an edge serves the stale (previously-valid-or-invalid) WAF config, with no signal — on the cluster or in metrics — that a given edge is behind the control plane for a given policy.

This surfaced while investigating #242: a tenant corrected an inverted-paranoia TPP upstream, but the edge continued serving the old inverted config (which 500s 100% of the route via CRS 901500 fail-closed) for a window before converging.

Evidence

Observed on prod us-central-1-alice, policy ns-c5f1cf4e…/kev1n-proxy:

Source	generation	paranoiaLevels
upstream (karmada control plane)	8	{detection: 2, blocking: 2} (valid — tenant-corrected)
edge us-central-1-alice (first read)	7	{detection: 1, blocking: 2} (inverted — 500s all traffic)
edge us-central-1-alice (later read)	8	{detection: 2, blocking: 2} (converged)

So the projection is eventually consistent — it did converge — but the edge served the broken inverted config for the lag window, and nothing exposed that the edge was a generation behind. This is not a stuck/broken sync; it's propagation latency with zero visibility.

Why it matters

• WAF config is failure-sensitive: an inverted-paranoia policy 500s 100% of a route (TrafficProtectionPolicy Enforce mode: 500 with truncated/empty body on all traffic #242, Mechanism A). Any remediation applied upstream has a silent tail where the edge is still broken.
• With ~15+ *-alice edges and 500+ project control planes, there is no per-edge/per-policy way to answer "is this edge serving the current generation of this TPP?"
• The same latency applies to any TPP change (mode flips Observe↔Enforce, ruleSet edits), not just paranoia corrections — cf. fix: retrigger EG translation on TPP spec changes #255 (retrigger EG translation on TPP spec change).

Asks

1. Observability: expose the applied TPP generation / resourceVersion per edge (metric or status), so control-plane vs edge skew is queryable. A spec.observedGeneration-style signal on the edge-side attachment, or a metric labeled by policy + edge, would let us alert on lag.
2. Bound / quantify the latency: measure typical and worst-case upstream→edge propagation time for a TPP spec change; decide whether it needs an SLO.
3. (stretch) Consider surfacing staleness at the control plane so a tenant/operator sees "not yet applied on N edges" after an edit.

Not this issue

• The upstream admission gate now rejects new inverted TPPs (self.detection >= self.blocking, verified live on karmada) — recurrence prevention is tracked separately.
• Already-persisted inverted TPPs / their neutralization via fix: flag inverted TrafficProtectionPolicy paranoia levels via status condition #252 — tracked in TrafficProtectionPolicy Enforce mode: 500 with truncated/empty body on all traffic #242.

Related: #242, #255.

[ecv]

Proposed regression-coverage acceptance

Propagation-staleness is invisible today and the parity gate (#225) only reads a single point in time. With the prod-fidelity env live (#275/#276) and policy delivery landing in #227, attach explicit coverage:

• A test/e2e/ scenario applies a TPP change upstream, then asserts the edge converges to the new generation and that convergence is observable — a spec.observedGeneration-style signal or per-edge/per-policy metric — not inferred from a request eventually succeeding.
• Scenario exercises the failure-sensitive path from TrafficProtectionPolicy Enforce mode: 500 with truncated/empty body on all traffic #242: correct an inverted-paranoia TPP upstream and assert the edge stops 500ing once (and only once) it reports the corrected generation.
• Guard asserts control-plane-vs-edge skew is queryable during the lag window (the observability ask in this issue), so an unbounded silent tail fails the test.

Depends on the observability signal from this issue's Ask #1 existing first — the test asserts against it.