Kubernetes operator for Celery

[jmdacruz]

This proposal is about having a Kubernetes operator (see here and here). The scope of the operator would be the following:

• Defining a CRD for a CeleryApplication. This resource would contain the configuration for the cluster (e.g., container resource requests/limits, number of replicas), Celery configuration (e.g., broker and result backend configuration), Docker image with the code and launch parameters (e.g., location of the code inside the container, virtualenv)
  • As an alternative, the CRD could include a URL for downloading a Python package with the code of the application, instead of the Docker image.
• The operator itself would manage the control loop of the CeleryApplication CRD, and would spawn a Kubernetes Deployment for the cluster, and also a Deployment for running flower. It would also create a Kubernetes Service so that we can access the flower UI/API
• Broker and result backend configuration would be out of scope of the operator, these have to be created before hand. A set of CeleryApplication resources should be able to use a shared broker and result backend, but they can pick their own broker configuration too.

This idea is inspired by the Flink Kubernetes Operator developed by Lyft: https://github.com/lyft/flinkk8soperator

[thedrow]

Yes, this should happen.
It means that we need to create a Controller implementation (See #19) for k8s.

[jmdacruz]

Initial attempt at implementing this (still very much a proof-of-concept): https://github.com/jmdacruz/celery-k8s-operator

[thedrow]

How do you imagine the integration with Celery 5 be like?

[jmdacruz]

I honestly haven't been following the development of 5.X that close, where can I get a glimpse on the biggest changes? Worst case scenario, if there are breaking changes, the operator supports versioning: The CeleryApplication resource includes an attribute for the celeryVersion (I'm currently ignoring it, but should be used for this) which can be used to change the shape of the Kubernetes deployment according to the Celery version.

[jmdacruz]

where can I get a glimpse on the biggest changes?

Actually, scratch that... you had already mentioned this above :-). I'll take a look at that.

[jmdacruz]

Trying to stick to the KISS principle, I think a good option would be to keep a single CRD (the CeleryApplication CRD) and a single Docker image for all the different tools and new "roles" (thanks to the celery CLI, the celery-k8s-operator uses a single image to run the workers, flower, and the liveness probes using celery inspect, something similar could be done to launch the Controller, Router, and Publisher?). The operator would create the deployment for the core components (e.g, the Celery Controller), ensuring that the proper configuration is injected (using Kubernetes ConfigMap) so that the Celery Controller can then do its job. Since the Celery Controller is now taking some of the responsibility the operator has for a 4.X deployment, then the operator needs to ensure the controller deployment/pods have the proper Kubernetes permissions to create the required resources.

Another approach would be to treat the Celery Controller as the operator itself, in which case its deployment should be straight forward (the expectation is that an operator is a standalone deployment), and then use the CeleryApplication CRD to create objects that will be impacted into the Kubernetes cluster by the Celery Controller operator.

In general, an Kubernetes operator needs to be very simple to deploy and maintain, since it's by definition a critical piece of the infrastructure (it becomes part of the Kubernetes cluster by extending its functionality, making sure other pieces work as expected)

Take a look at what the folks at Lyft do for the Apache Flink operator: https://github.com/lyft/flinkk8soperator

[gautamp8]

Is this thread still active? Inspired by the need for this at my own work, I also tried building a POC/MVP of Celery operator to learn the whole thing - https://github.com/brainbreaker/Celery-Kubernetes-Operator

I also presented it in EuroPython 2020 last Friday as part of my talk around automating the management of Kubernetes Infra while staying in the Python ecosystem(Slides - https://bit.ly/europython20-ppt). I'm willing to commit a certain number of hours every week to build a production-ready version of the Celery operator.

[mik-laj]

In the Apache Airflow project, we use KEDA to provide autoscaling from 0 to n.
https://www.astronomer.io/blog/the-keda-autoscaler/

[auvipy]

In the Apache Airflow project, we use KEDA to provide autoscaling from 0 to n.
https://www.astronomer.io/blog/the-keda-autoscaler/

I quite liked the approach KEDA take when I first saw that.

[thedrow]

I'm aware of KEDA and I plan to incorporate it in our solution.

[gautamp8]

Yes, KEDA is probably the best way to go for scaling use-case. It keeps us close to native solutions like HPA and only introduces the metrics server and controller.

For my application, I was personally more focused around the learning experience so chose to implement a really basic scaling algorithm without using anything external.

[thedrow]

What other tasks should the operator perform other than scaling?