WAF response-body buffering: proxy memory scales with concurrent inspected responses

[ecv]

Behavior

When a TrafficProtectionPolicy runs in Enforce mode, the coraza filter inspects the response phase, which means Envoy must buffer the response body in memory until coraza finishes evaluating the phase-4 (RESPONSE) rules before any bytes are released downstream. Buffered bytes are held per in-flight connection, so the proxy's memory scales with the number of concurrent responses being inspected multiplied by the size of each buffered body — a dimension that does not show up at rest at all.

At rest the proxy is small. Measured on the local WAF repro (single downstream Envoy, coraza .so + CRS loaded, no traffic): the envoy container sits at ~181 MB RSS, and the whole WAF cluster is ~1.76 GB. Under load the same proxy can climb into multiple gigabytes purely from held response bodies, which is where the "~15 GB during a WAF e2e" observation came from — it was buffering under traffic, not a steady footprint.

Two hard edges interact here:

• SecResponseBodyLimit bounds how much of a single response coraza will buffer, and it must stay at or below the Envoy per_connection_buffer_limit_bytes (32768 in the repro stack). If a held/buffered response exceeds that, the request fails with a 500 (response_payload_too_large) rather than degrading gracefully.
• Aggregate memory is unbounded by those per-connection limits alone: concurrent_inspected_responses × per_response_buffer is what actually sizes the proxy, and nothing in the current config caps that product.

Why it matters

• OOM risk / noisy-neighbor: a burst of concurrent large responses can push a proxy pod to its memory limit and get it OOM-killed, taking out unrelated routes sharing the proxy.
• Latency: full-body buffering delays time-to-first-byte for every inspected response, worst for large or streamed payloads.
• Surprise 500s: the response_payload_too_large failure mode is easy to trip and hard to diagnose without knowing the limit relationship.
• Capacity planning gap: proxy memory requests/limits today are not derived from any concurrency × buffer model, so they are guesses.

Deliverable

A committed doc (under docs/) capturing:

1. The behavior — response-phase buffering, the SecResponseBodyLimit ↔ per_connection_buffer_limit_bytes relationship, the response_payload_too_large failure mode, and the at-rest vs under-load memory numbers.
2. A scaling / capacity model — peak proxy memory ≈ baseline + (peak concurrent inspected responses × per-response buffer), with worked numbers, so pod memory requests/limits and replica counts can be sized rather than guessed.
3. Mitigations (evaluate and recommend):
   • Cap SecResponseBodyLimit tightly and keep it ≤ per_connection_buffer_limit_bytes.
   • Skip response-body inspection for large / streamed content-types (downloads, media) or specific routes, so big bodies never buffer.
   • Set per-proxy memory limits and requests derived from the model, plus a PDB, to bound blast radius.
   • Autoscale proxies on memory, not just CPU.
   • Prefer detection-only for the response phase where enforcement is not required.
   • Return a clean 413 rather than a 500 when a body exceeds the inspection limit.

Repro to quantify

Drive the /leak response-body path in the WAF repro under increasing concurrency and body size, and record proxy RSS vs concurrency to fit the model. Stack and steps: hack/waf-repro/ (see the WAF local repro notes).

[ecv]

hmmm not reliably reproducing.