Limit deadline alerts endpoint to the latest serialized Dag row - #70804
Merged
Conversation
ordering all rows for a dag_id descending and reading the first one. Without a LIMIT, the database materializes every serialized version of the Dag when only the newest is needed. Cap the lookup at a single row.
imrichardwu
requested review from
bugraoz93,
choo121600,
ephraimbuddy,
henry3260,
jason810496,
pierrejeambrun,
rawwar and
shubhamraj-git
as code owners
July 31, 2026 01:31
eladkal
approved these changes
Jul 31, 2026
eladkal
reviewed
Jul 31, 2026
eladkal
added
type:bug-fix
ferruzzi
approved these changes
Jul 31, 2026
ferruzzi
left a comment
ferruzzi
left a comment
Contributor
There was a problem hiding this comment.
Left a non-blocking suggestion in Elad's comment thread, otherwise LGTM
A backfill run executes the Dag version that was current when it was queued, whose deadline alerts may differ from the ones deployed today. The endpoint could only ever resolve the newest serialized Dag, so the UI reported present-day alerts for historical runs and misrepresented what those runs were actually held to. Resolving the version through DagVersion.version_number also removes a reliance on the serialized_dag primary key sorting chronologically, which only held because uuid7 happens to be time-ordered and would give the wrong row if an older version were serialized after a newer one. Only the primary key is read, so the serialized Dag blob no longer crosses the wire for a lookup that discards it.
imrichardwu
requested review from
bbovenzi,
guan404ming,
ryanahamilton and
vatsrahul1001
as code owners
August 1, 2026 01:29
Contributor
Author
|
I updated it based on your suggestion @ferruzzi |
Contributor
Backport successfully created: v3-3-testNote: As of Merging PRs targeted for Airflow 3.X In matter of doubt please ask in #release-management Slack channel.
|
bbovenzi
pushed a commit
that referenced
this pull request
Aug 3, 2026
…ag row (#70804) (#70965) * The deadline alerts endpoint fetches the latest serialized Dag by ordering all rows for a dag_id descending and reading the first one. Without a LIMIT, the database materializes every serialized version of the Dag when only the newest is needed. Cap the lookup at a single row. * Sort imports in deadline alerts UI route tests * UI: Allow reading deadline alerts of a specific Dag version A backfill run executes the Dag version that was current when it was queued, whose deadline alerts may differ from the ones deployed today. The endpoint could only ever resolve the newest serialized Dag, so the UI reported present-day alerts for historical runs and misrepresented what those runs were actually held to. Resolving the version through DagVersion.version_number also removes a reliance on the serialized_dag primary key sorting chronologically, which only held because uuid7 happens to be time-ordered and would give the wrong row if an older version were serialized after a newer one. Only the primary key is read, so the serialized Dag blob no longer crosses the wire for a lookup that discards it. (cherry picked from commit 1b3121d) Co-authored-by: Richard <richard9@ualberta.ca>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The
GET /ui/dags/{dag_id}/deadlineAlertsendpoint queriesserialized_dagfor the latest serialized DAG by ordering rows in descending order, but the query never applies aLIMIT. Althoughsession.scalar()returns a single object, it does not limit the SQL query itself. The database and ORM still materialize the full result set before discarding every row except the first.Since every
serialized_dagrow contains the full serialized DAG blob, this becomes a serious scalability issue for DAGs with long version histories. A DAG with thousands of historical versions can force the API server to load gigabytes of serialized data into memory just to return the newest entry. In one reported case, a DAG with roughly 6,725 versions caused the API process to consume about 4 GiB of memory and exceed its memory limit.Adding
.limit(1)ensures the database returns only the newest row. This reduces memory usage from scaling with a DAG's version history to a constant amount, while also allowing the query planner to stop after the first matching index entry instead of processing every matching row.This also makes the implementation consistent with the existing
_get_latest_serdaghelper ingrid.py, which already performs the same lookup withLIMIT 1.closes: #70779
Was generative AI tooling used to co-author this PR?
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.