Skip to content

Fix: Use a short-live session instead of passing sessionDeps into DagRunWaiter#64520

Draft
sjyangkevin wants to merge 1 commit intoapache:mainfrom
sjyangkevin:fix-dag-wait-endpoint-session
Draft

Fix: Use a short-live session instead of passing sessionDeps into DagRunWaiter#64520
sjyangkevin wants to merge 1 commit intoapache:mainfrom
sjyangkevin:fix-dag-wait-endpoint-session

Conversation

@sjyangkevin
Copy link
Copy Markdown
Contributor

@sjyangkevin sjyangkevin commented Mar 31, 2026

related: #64415

Why

The SessionDep session is held open from the moment the endpoint function starts until the StreamingResponse generator is fully consumed and FastAPI cleans up the dependency. During the polling loop (while dag_run.state not in State.finished_dr_states: await asyncio.sleep(self.interval)), the sync DB connection sits idle but checked out from the pool. If a DAG run takes minutes or hours to complete, that connection is locked for the entire duration. With many concurrent /wait requests, this can exhaust the connection pool.

This PR improves the above by opening a short-live session in _serialize_xcoms, similar to what have been done in _get_dag_run.

#64415 (comment)

Test Cases

Tests should have been handled in test_collect_task and test_should_respond_200_immediately_for_finished_run


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: [Claude Code Opus 4.6] following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@boring-cyborg boring-cyborg bot added the area:API Airflow's REST/HTTP API label Mar 31, 2026
@sjyangkevin sjyangkevin changed the title use a short-live session instead of passing sessionDeps into DagRunWaiter Fix: Use a short-live session instead of passing sessionDeps into DagRunWaiter Mar 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:API Airflow's REST/HTTP API

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants