Add Prometheus Metrics for Observability

[cstanislawski]

Currently, the lifecycle-controller operates without exposing any metrics. This makes it difficult to monitor its health, understand its workload (how many resources it's managing), and diagnose issues without parsing logs.

We should expose a /metrics endpoint to provide key metrics in the Prometheus format. This will allow for easy integration with standard Kubernetes monitoring stacks.

Some initial metrics that we could provide:

• lifecycle_controller_watched_resources: A Gauge to show the current number of resources being managed by the controller

  • Labels: action_type (e.g., delete, restart), annotation (e.g., lifecycle.io/delete-at, lifecycle.io/restart-cron)

• lifecycle_controller_actions_total: A Counter to track the number of actions performed

  • Labels: action_type (e.g., delete, restart), status (success, failure), dry_run (true, false)

• lifecycle_controller_errors_total: A Counter for specific misconfigurations detected

  • Labels: reason (conflicting_annotations, invalid_timezone)

• lifecycle_controller_reconcile_duration_seconds: A Histogram to measure the latency of the reconciliation loop, helping to identify performance issues