Skip to content

Add indexes on span_status and dag_version_id in task_instance#53746

Open
anandkaranubc wants to merge 2 commits intoapache:mainfrom
anandkaranubc:migration/53401
Open

Add indexes on span_status and dag_version_id in task_instance#53746
anandkaranubc wants to merge 2 commits intoapache:mainfrom
anandkaranubc:migration/53401

Conversation

@anandkaranubc
Copy link
Copy Markdown
Contributor

@anandkaranubc anandkaranubc commented Jul 25, 2025

closes: #53401

This PR adds two indexes to the task_instance table to improve query performance:

  • idx_span_status on the span_status column
  • idx_dag_version_id on the dag_version_id column

These indexes help speed up scheduler and DAG processor operations when the task_instance table has a large number of records (~10M+), as noted in issue #53401.


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@anandkaranubc anandkaranubc changed the title Add indexes on span_status and dag_version_id in task_instance Add indexes on span_status and dag_version_id in task_instance Jul 25, 2025
@potiuk
Copy link
Copy Markdown
Member

potiuk commented Jul 25, 2025

Well. You also need to change definition of the models - migrations generally should be generated as result of changes in the models as far as I understand.

@amoghrajesh
Copy link
Copy Markdown
Contributor

Just linked the PR to the issue: closes: #53401

@AbhiSinha08
Copy link
Copy Markdown

I am working on adding the indexes manually on DB in my Airflow 3 instance since it is affecting DP performance. I am curious why we are having id as the first column in the index idx_span_status

@github-actions
Copy link
Copy Markdown

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

@github-actions github-actions bot added the stale Stale PRs per the .github/workflows/stale.yml policy file label Feb 17, 2026
@potiuk
Copy link
Copy Markdown
Member

potiuk commented Mar 2, 2026

@anandkaranubc We are unassigning you from this issue as part of our updated assignment policy.

This is not meant to discourage your contribution — quite the opposite! You are still very welcome to work on this issue and submit a PR for it. Simply comment that you are working on it and open a PR when ready.

We found that formal assignments were not working well, as they often prevented others from contributing when the assignee was not actively working on the issue.

@github-actions github-actions bot removed the stale Stale PRs per the .github/workflows/stale.yml policy file label Mar 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

create a new index for span_status, dag_version_id in task_instance table

7 participants