e2e suite flakes on a cold CI env (readiness races fixed timeouts)

[ecv]

Symptom

On a freshly brought-up env (the first tests to run, before the stack is warm), tests that create a Gateway or a Domain intermittently time out against fixed per-test deadlines. Observed while landing CI-on-env (#276):

• gateway-invalid-certificate FAILED on run [29357597376] — Post .../domains: context deadline exceeded (NSO validating webhook not yet serving) — then PASSED on re-dispatch [29358643832] in 25.98s (vs 10.79s when it failed). Pure flake.
• billing-http-metering fails cold in CI (No Envoy access log found containing 'httproute/', or the downstream Gateway Programmed assert times out at 30s). On a warm env it passes end-to-end (verified locally, 61.68s: access log emitted, project_name injected, all four CloudEvents delivered). So the code is correct — it's a cold-start race.

Root cause

test-infra:wait-ready only waits for the controller-manager Deployment to be Available. That is not sufficient:

• the validating/defaulting webhook may not be serving yet (cert mounted + endpoint ready), so early Domain/gateway creates hit context deadline exceeded;
• the downstream Envoy Gateway + ext-server may still be warming, so the first Gateway takes >30s to reach Programmed and the data-plane access log isn't tail-able yet.

Fresh-env cold start races the tests' fixed timeouts (30s Programmed assert, 45s billing scrape).

Fix options

• Harden wait-ready: block until the NSO webhook actually admits (server-side dry-run of a throwaway object), and until the downstream GatewayClass/EG is serving — not just Deployment Available.
• And/or bump the tight per-test timeouts (downstream Programmed assert; billing connectivity/scrape).
• Optionally a warm-up gateway before the suite so the merged proxy exists.

Repro

Dispatch E2E Tests on test/prod-fidelity-env-rebased from cold; the earliest gateway/domain test may flake. Re-dispatch usually passes.

Related

• test(infra): prod-fidelity edge test env + e2e suite #276 (CI on the prod-fidelity env), Lift the e2e suite onto the prod-fidelity two-cluster env (#226) #275