feat(observability): add OpenTelemetry tracing and Prometheus metrics

[kiro-agent]

This pull request was created by @kiro-agent on behalf of @ccrsxx 👻

Comment with /kiro fix to address specific feedback or /kiro all to address everything.
_{Learn about Kiro Web}

---

Stacked on #30 — targets feat/observability-logging-foundation, so the diff shows only the metrics + tracing work. Merge #30 first.

Roadmap Steps 5 & 6. This is the part that gives you the Cloud Trace-style waterfall and the CPU/RAM/goroutine visibility.

Tracing

• internal/observability/tracing.go — OTLP HTTP exporter, resource attributes (service.name / service.namespace / deployment.environment.name), ParentBased(TraceIDRatioBased) sampler, W3C trace context + baggage propagation.
  • No-op when disabled (TRACING_ENABLED=false or empty endpoint) so local dev and tests need zero infrastructure.
  • Telemetry failures never take down the API — export errors are logged as WARN via a custom otel.ErrorHandler.
• HTTP server wrapped with otelhttp. Spans are named after the matched mux route, not the raw path — the default formatter would create one span name per blog slug and blow up Tempo's span-name index. /sse is excluded (multi-hour spans).
  • Because http.ServeMux only populates r.Pattern while dispatching, a TraceRoute middleware sits innermost and renames the span + sets http.route once the route is known.
• Database: otelpgx on the pool → a span per SQL query.
• Outbound clients: new utils.NewHTTPClient adds an otelhttp transport; applied to all 7 HTTP clients (cloudflare, github, jellyfin, navidrome, pixiv, pushover, spotify). A slow Spotify now shows up as a labelled span instead of unexplained latency.
• RequestID now prefers the active span's trace/span IDs, so trace_id in the logs is the same one Tempo stores (this is what makes the log → trace jump work). Falls back to traceparent parsing when tracing is off.

Metrics

• Dedicated Prometheus registry (not the global default) with Go runtime + process collectors — go_goroutines is the key leak signal here, since every SSE client parks a goroutine and cache writes are fire-and-forget.
• http_requests_total and http_request_duration_seconds, labelled by matched route only.
• App metrics that were previously invisible: sse_active_clients, cache_operations_total{result}, and upstream_request_duration_seconds{platform,result}.
  • The upstream one matters most: getSSEData silently falls back to a default payload on error, so an outage was indistinguishable from "nothing is playing". The three duplicated fetch closures are now one instrumented helper.
• /metrics and /healthz on a separate internal-only port (METRICS_PORT, default 9090), deliberately not on the public router and not published in compose — the collector scrapes it over the observability network. /healthz pings the pool.

Wiring / ops

• Init order is now explicit in main.go: loaders → tracing → pool → handlers. This matters: otelpgx and otelhttp capture the global tracer provider at construction time, so initializing tracing later would silently lose every span.
• Spans are flushed last during graceful shutdown, inside the existing 60s window.
• compose: joins the shared external observability network, sets GOMEMLIMIT=450MiB alongside a 512M memory limit (a container limit without GOMEMLIMIT means Go gets OOM-killed instead of GCing harder).

Testing

• go build ✅ · go vet ✅ · gofmt ✅ · go test ./... ✅ 31/31 packages pass, 0 failures (no existing test changed)
• Runtime smoke-tested (throwaway harness, not committed):
  • /healthz → 200 "ok"; /metrics → 200, 12.5 KB, all expected series present incl. Go runtime.
  • Cardinality verified: 3 requests to /contents/a|b|c produced exactly one series, route="GET /contents/{slug}" — no raw slugs leaked into labels.
  • Graceful degradation verified: with tracing enabled pointing at an unreachable collector, init returned instantly, the request still returned 200 with X-Request-Id set, the export failure logged as WARN, and shutdown was clean.
  • Correlation verified: in-handler trace_id was a real 32-char OTel trace ID from the active span, not a generated placeholder.
• Note: golangci-lint still can't run in the sandbox (binary built with Go 1.24 < this repo's 1.26 target). Please confirm CI lint.

To try it out

docker network create observability   # once
# in .env
TRACING_ENABLED=true
OTEL_EXPORTER_OTLP_ENDPOINT=http://alloy:4318

Then uncomment the prometheus.scrape "api" block in docker/observability-agent/alloy/config.alloy (homelab#5) to start scraping api:9090.

With tracing left off, this PR is inert apart from the new /metrics port — safe to merge and enable later.