As the controller manages more resources, it could potentially create a "thundering herd" problem, overwhelming the Kubernetes API server if many actions are scheduled for the same time.
We should introduce controls to manage load and ensure the controller operates safely and efficiently at scale.
We can add the following features:
-
Max Concurrent Reconciles - A controller-level flag to limit the number of resources being processed in parallel. This protects the controller and the API server from being overloaded.
-
Action Jitter - A per-resource annotation, lifecycle.io/action-jitter: "5m", to add a random delay before executing a scheduled action. This helps spread out the load when many resources share the same schedule (e.g., a nightly restart).
As the controller manages more resources, it could potentially create a "thundering herd" problem, overwhelming the Kubernetes API server if many actions are scheduled for the same time.
We should introduce controls to manage load and ensure the controller operates safely and efficiently at scale.
We can add the following features:
Max Concurrent Reconciles - A controller-level flag to limit the number of resources being processed in parallel. This protects the controller and the API server from being overloaded.
Action Jitter - A per-resource annotation,
lifecycle.io/action-jitter: "5m", to add a random delay before executing a scheduled action. This helps spread out the load when many resources share the same schedule (e.g., a nightly restart).