KubernetesExecutor scheduler crashes with PicklingError on pod_override with kubernetes client 36.x

[vatsrahul1001]

Apache Airflow Provider(s)

cncf-kubernetes

What happened

With KubernetesExecutor running in-cluster, any task that sets a V1Pod pod_override in executor_config crashes the scheduler in a loop:

_pickle.PicklingError: Can't pickle local object
  <function InClusterConfigLoader._set_config.<locals>._refresh_api_key at 0x...>
when serializing kubernetes.client.configuration.Configuration ...
when serializing dict item 'pod_override' ... ExecuteTask ...
  File ".../executors/kubernetes_executor.py", line 230, in execute_async
    self.task_queue.put(KubernetesJob(key, command, kube_executor_config, pod_template_file))

execute_async puts the KubernetesJob (embedding the task's pod_override V1Pod) onto a multiprocessing.Manager queue, whose .put() pickles synchronously in the scheduler process — so a pickle failure takes the scheduler down rather than failing the task.

Root cause

In kubernetes-client/python 36.0.0, the generated model __init__ changed how the default local_vars_configuration is set (PR kubernetes-client/python#2532, upgrading to OpenAPI Generator v6.6.0 — OpenAPITools/openapi-generator#8500):

# kubernetes 35.x
if local_vars_configuration is None:
    local_vars_configuration = Configuration()            # empty -> picklable

# kubernetes 36.x
if local_vars_configuration is None:
    local_vars_configuration = Configuration.get_default_copy()   # copy of the in-cluster default

In-cluster, the default Configuration carries refresh_api_key_hook = InClusterConfigLoader._set_config.<locals>._refresh_api_key, a local closure that pickle cannot serialize. So in 36.x every in-cluster V1Pod inherits an unpicklable Configuration.

How to reproduce

In-cluster KubernetesExecutor, kubernetes client 36.x, run a task with a V1Pod pod_override:

from kubernetes.client import models as k8s

PythonOperator(
    task_id="t",
    python_callable=lambda: None,
    executor_config={
        "pod_override": k8s.V1Pod(
            spec=k8s.V1PodSpec(
                containers=[
                    k8s.V1Container(
                        name="base",
                        resources=k8s.V1ResourceRequirements(
                            requests={"cpu": "100m", "memory": "384Mi"},
                        ),
                    )
                ]
            ),
        )
    },
)

Operating System

Any (Linux, in-cluster). Reproduced on Astro Runtime with Airflow 3.2.x and 3.3.x.

Versions of Apache Airflow Providers

apache-airflow-providers-cncf-kubernetes==10.18.0, kubernetes==36.0.0 / 36.0.2, in-cluster KubernetesExecutor.

Deployment

Official Apache Airflow Helm Chart / Astronomer

Deployment details

In-cluster KubernetesExecutor (scheduler loads config via load_incluster_config).

Proposed fix

#68819 — serialize the pod_override to a plain dict before queuing (dropping the Configuration) and rebuild the V1Pod worker-side, so the executor is robust regardless of the kubernetes client version. The worker side already sanitizes pods with sanitize_for_serialization; this closes the same gap on the scheduler -> queue boundary. The underlying client behavior should also be reported upstream to kubernetes-client/python (root cause: #2532 / openapi-generator#8500).

Are you willing to submit PR?

• Yes I am willing to submit a PR! (Fix KubernetesExecutor scheduler crash when pod_override is queued in-cluster #68819)

Code of Conduct

• I agree to follow this project's Code of Conduct

---

Drafted-by: Claude Code (Opus 4.8); reviewed by @vatsrahul1001 before posting