feat(telemetry): alert on gateway reconcile-error ratio#282
Open
ecv wants to merge 1 commit into
Open
Conversation
Add two alerts on the gateway controller's reconcile error ratio so a controller that cannot apply configuration pages operators directly instead of failing silently. In production after v0.23.4 the controller ran at an ~83% reconcile failure rate for hours with nothing firing; cert-listener withholding can empty a downstream Gateway's listeners, which the CRD rejects, so the controller hot-loops and stale config stays programmed at the edge. Rebase of #218 onto current main, with the runbook anchor typo (errorrato -> errorratio) corrected in the alert and test fixtures so the runbook_url deep-links resolve. Key changes: - Append GatewayControllerReconcileErrorRatioHigh (warning, >20% for 15m) and GatewayControllerReconcileErrorRatioCritical (critical, >50% for 10m) to config/telemetry/alerts/gateways.yaml - Add docs/runbooks/gateway-controller-health.md with impact, diagnosis, and remediation for each tier - Add promtool unit tests covering firing at each threshold, not firing below it, and not firing before the for window elapses
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rebase of #218 onto current
main(the original was 129 commits behind), plus a one-word anchor fix. Draft for review.What
Two alerts on the gateway controller's reconcile error ratio, so a controller that can't apply configuration pages operators directly instead of failing silently:
Both link a new runbook. In production after v0.23.4 the controller ran at ~83% reconcile failure for hours with nothing firing (cert-listener withholding empties a downstream Gateway's listeners, the CRD rejects it, the controller hot-loops, stale config stays programmed). These would have fired immediately.
Safety
gateways.yaml.Named("gateway");controller_runtime_reconcile_totalis standard controller-runtime; no divide-by-zero (the error series is a subset of the total).Difference from #218
The original's
runbook_urlanchors had a typo —#gatewaycontrollerreconcileerr**ato**high/...erratocritical— that didn't match the runbook headers (...ErrorRatioHigh/Critical), so the deep-links broke. Fixed toerrorratio*in the alert and both test fixtures.Verification
promtool check rules— SUCCESS (2 rules);promtool test rules— SUCCESS (the CItest-prometheus-rulestask auto-globs the fixture).kustomize build config/telemetry/alerts— OK.Supersedes #218.