Airflow scheduler memory spikes when a dag is run with large run config

[deepak4babu]

Under which category would you file this issue?

Airflow Core

Apache Airflow version

3.2.0

What happened and how to reproduce it?

When a dag with dynamic task is triggered with large run config, the scheduler memory increases by 5-6 times the regular usage when scheduling the dynamic tasks. Once the dagrun completes, the memory usage is back to normal.

The run config is a json of size 512KB to 2MB

To reproduce, create a dag with dynamic task that create 500+ tasks. Trigger the dag with a json of size 512KB to 1MB as run config

What you think should happen instead?

The scheduler memory should not spike when running dynamic task with large run config.

Operating System

Debian GNU/Linux 12 (bookworm)

Deployment

Other 3rd-party Helm chart

Apache Airflow Provider(s)

No response

Versions of Apache Airflow Providers

apache-airflow-providers-celery==3.17.1
apache-airflow-providers-cncf-kubernetes==10.14.0
apache-airflow-providers-common-compat==1.14.1
apache-airflow-providers-common-io==1.7.1
apache-airflow-providers-common-sql==1.33.0
apache-airflow-providers-fab==3.6.1
apache-airflow-providers-smtp==2.4.3
apache-airflow-providers-standard==1.12.1

Official Helm Chart version

Not Applicable

Kubernetes Version

Not Applicable

Helm Chart configuration

Not Applicable

Docker Image customizations

Not Applicable

Anything else?

Issue happens in dynamic dags, everytime, the dag is triggered with big config.

Older issue for reference - #49076

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct