[EPIC] Production-grade observability & runtime reliability (mainnet blocker)

[joelpeace48-cell]

Epic / consolidation. This merges several smaller observability & reliability tasks into one high-priority, mainnet-blocking initiative. Supersedes #563, #570, #571, #572, #576, #577, #579.

Why this matters (growth/scale)

A platform meant for thousands of users cannot grow on top of a system that's blind to its own failures. Reliability is a growth feature: every outage or silent failure during a high-traffic campaign launch burns user trust and operator confidence. This initiative makes Trivela observable and self-defending so it can scale without surprise outages.

Goal

Ship production-grade observability + runtime reliability: dashboards, alerts, a live canary, request deadlines, pool/saturation visibility, and graceful shutdown — wired to SLOs.

Scope (merged work items)

• Metrics & dashboards — commit Grafana dashboards as code (observability/dashboards/*) for API, DB/pools, RPC/breaker, and indexer; provisioned + reproducible. (was feat: Grafana dashboards as code (API, DB, RPC, indexer, KPIs) #577, feat: Indexer observability dashboard (lag, throughput, errors) #563)
• Alerting — Prometheus alert rules (5xx rate, p95 latency, RPC degradation/breaker-open, indexer lag, pool saturation, operator low-balance, DLQ growth) with promtool tests + routing. (was feat: Prometheus alerting rules (latency, errors, RPC, indexer lag, pools) #576)
• Synthetic canary — scheduled register→credit→claim canary on testnet + uptime checks; alert within minutes on failure (also flags stale README contract IDs). (was feat: Synthetic uptime + register→credit→claim canary monitoring #579)
• Request deadlines — per-route timeouts + AbortSignal propagation to DB/RPC/storage; abort on client disconnect; 504 on deadline. (was feat: Per-route request timeouts, deadlines & cancellation propagation #570)
• Pool saturation safety — export pool_in_use/idle/waiting metrics; acquire timeouts; typed 503 POOL_SATURATED instead of hanging. (was feat: Connection-pool saturation metrics & fast-fail safeguards #571)
• Graceful shutdown — drain HTTP, checkpoint in-flight jobs, close SSE/WS with reconnect hint within a grace window. (was feat: Graceful shutdown for in-flight jobs & SSE/WS streams #572)
• SLOs — define availability/latency/indexer-freshness SLIs + error budget that the alerts derive from.

Acceptance criteria

• Dashboards render against live metrics; alerts fire on synthetic breaches (promtool tests pass).
• A broken core journey is detected by the canary within minutes.
• Slow upstreams return timely 504s; saturated pools fast-fail with a typed 503; rolling deploys drop no work.

Verification

• promtool test rules in CI; force a canary failure; load test driving pool saturation; SIGTERM-under-load deploy test.

Priority: high · Difficulty: hard · Effort: L · mainnet blocker

[Menjay7]

Hi! I came across this issue and would love to work on it. I have experience with similar tasks and I'm confident I can investigate, implement a clean solution, and thoroughly test it before submitting a PR. If the issue is still open, I would appreciate it if you could assign it to me. Thank you!