[FEATURE] Consider richer health signals for /livez and /readyz

[sunib]

This is an observation from reading the code, not a known bug or a reproduced failure. The probes work as written and the trade-offs the code makes (sticky readiness on requestheader trust, no functional path check) look deliberate. I'm filing it as a discussion / FR because a richer signal would have been useful in a specific scenario this week, and it's the kind of "would be nice" that comes up more than once in conversations.

Is your feature request related to a problem? Please describe.

/livez and /readyz today reflect "the HTTP server is up" and "the requestheader trust snapshot loaded once at startup" respectively. Neither reflects whether the data path is actually working.

cmd/server/main.go:472-489:

func handleHealth(w http.ResponseWriter, _ *http.Request) {
    w.WriteHeader(http.StatusOK)
    _, _ = w.Write([]byte("ok"))
}

func readinessHandler(controller *identity.RequestHeaderTrustController) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
        if !controller.Ready() {
            http.Error(w, "requestheader trust not ready", http.StatusServiceUnavailable)
            return
        }
        w.WriteHeader(http.StatusOK)
        _, _ = w.Write([]byte("ok"))
    })
}

The comment above readinessHandler is explicit that the trust check intentionally never flips back: "A transient loss of the watch does not flip it back: the controller retains last-known-good trust." That part feels correct — readiness shouldn't oscillate on a transient ConfigMap watch hiccup.

What it leaves uncovered is the rest of the data path:

• Backend reachability (backend.url returning 5xx every request).
• Webhook delivery success ratio over a recent window.
• Whether the proxy has actually successfully forwarded anything recently when traffic is present.

Scenario where richer readiness would have helped (observed, not reproduced for this report):

The receiver-side (gitops-reverser) pod entered a restart loop. While its gitops-reverser-audit Service had zero endpoints, the audit-proxy logged a continuous stream of best-effort webhook delivery failed ... Operation not permitted errors and dropped enriched audit bodies. From kubelet's point of view, the audit-proxy pod kept reporting 1/1 Ready the entire time. The audit-proxy was, in the strict TCP/TLS sense, healthy. In the "I am doing my job" sense, it wasn't — every audit event during that window was lost (the proxy is best-effort, by design, so the behaviour is right; the signal is what's missing).

To be clear: this is not necessarily a bug — best-effort delivery is a deliberate design choice, and a stale-but-serving proxy is often better than a flapping one. The ask is just: surface this state somewhere, so operators (and Kubernetes itself) can react.

Describe the solution you'd like

A richer set of signals, with the trade-offs surfaced as configurable knobs rather than hard-coded behaviour. Some plausible shapes, listed roughly from least to most aggressive:

1. Metrics first, probes unchanged. Expose Prometheus-style counters/gauges for:
   • audit_proxy_webhook_deliveries_total{outcome="success|failure"}
   • audit_proxy_backend_responses_total{outcome="success|failure", status_class="2xx|4xx|5xx"}
   • audit_proxy_last_successful_webhook_timestamp_seconds
   • audit_proxy_last_inbound_request_timestamp_seconds
   • audit_proxy_requestheader_trust_age_seconds
   • This is the lowest-risk option: operators alert on it via VictoriaMetrics / Prometheus, no kubelet behaviour change.
2. /healthz (or a new /statusz) returning structured JSON with the same signals as (1) plus the requestheader-trust freshness, for human inspection / kubectl exec / curl. Probes still pass on TLS-listener-up.
3. Optional "functional readiness" mode, off by default. A chart value like server.readiness.requireRecentDelivery: { enabled: false, window: 60s, minSuccessRatio: 0.5 } that, when enabled, flips /readyz to NotReady when there has been recent inbound traffic but no successful webhook delivery in the configured window. Crucially gated by "has there been recent inbound traffic" so an idle proxy with no clients doesn't false-fail.
4. Same shape as (3) for backend reachability. Even softer trade-off: a single 5xx burst from the backend shouldn't necessarily cull all proxy replicas — readiness would gate on persistent failure over a window.

I'd lean toward (1) + (2) as the safe core, and (3)/(4) as opt-in modes for operators who want kubelet-level reaction. The chart's three-replica deployment shape (we just bumped to 3) is where (3) starts to make sense: with N replicas, occasional readiness flapping is absorbable.

Describe alternatives you've considered

• Leave the probes as-is. Defensible — they prove the listener is up and the trust snapshot loaded. Operators monitor data path health out-of-band via metrics or audit-stream completeness on the receiver. This is the status quo and is fine; it's just a missing convenience.
• Aggressive /livez that pings the backend. Rejected for myself before suggesting it: a transient cozystack-api blip would liveness-fail every proxy replica simultaneously, which is strictly worse than "stale but serving." /livez should stay process-liveness only.
• Restart-on-stuck via a sidecar / cron probe. Pushes the policy out of the proxy entirely. Workable but operators would each build their own; baking a tested version into the chart is friendlier.

Additional context

This is genuinely a "would be nice" rather than a "this is broken." The probes work as documented; the discoverable failure mode is "the receiver was down and the proxy didn't tell anyone besides the log stream." Combined with the upstream effort already invested in informative startup logging (the staged "first inbound request / first verified identity / first backend response / first audit event delivered" log lines were great when we ran the first install), surfacing runtime health in a similar shape feels like a natural next step.

Sibling-shaped to a few feature requests filed against the receiver
(ConfigButler/gitops-reverser#145,
#146,
#147,
#148) —
all around the same theme of "the data path works, the operational lifecycle around it could be a bit richer."

Implementation considerations

• Metrics counters can live in the existing instrumentation surface (Prometheus on the same port as /metrics if there is one, or a new port wired through the chart).
• For (3), the "has there been recent inbound traffic" gate is what avoids the obvious idle-proxy footgun. Without it, a proxy with no traffic looks unhealthy by definition.
• Sticky-with-staleness for requestheader trust is worth keeping; consider exposing requestheader_trust_age_seconds so operators can alert on "trust hasn't been refreshed in N minutes" without involving probe behaviour.
• Chart values should make the probes' behaviour configurable per-environment — sensible defaults that match today's behaviour, with the richer modes opt-in.

Priority

• Low
• Medium / High / Critical

(Low: there's a working pattern today, the gap is observability-shaped rather than functional, and richer probe behaviour carries its own foot-guns that warrant deliberate design rather than a quick patch. This is more "discussion to have" than "ship next sprint.")