Skip to content

Rename dag serialization metrics. - #70838

Open
Ei-Sandi wants to merge 1 commit into
apache:mainfrom
Ei-Sandi:rename-dag-serialisation-metrics
Open

Rename dag serialization metrics.#70838
Ei-Sandi wants to merge 1 commit into
apache:mainfrom
Ei-Sandi:rename-dag-serialisation-metrics

Conversation

@Ei-Sandi

Copy link
Copy Markdown
Contributor

Description

Follow-up to #68906, which added a single dag.serialization_writes counter emitted from two different branches of SerializedDagModel.write_dag. Sharing one name means the metric can't distinguish the two ways a Dag gets serialized, and they have very different consequences:

  • a new DagVersion row is inserted — unbounded growth across dag_version, dag_code and serialized_dag
  • the latest DagVersion is overwritten in place — bounded churn on a single row

A Dag that re-serializes on every parse (for example one whose template holds a callable, with no actual change to the Dag) shows up in either path depending on whether its current version has task instances, but only the first grows the metadata DB. Splitting the counter lets the two be alerted on with different thresholds.

dag.serialization_writes is replaced by:

metric emitted when
dag.serialization.version_created a new Dag version is written — first serialization, or the latest version already has task instances
dag.serialization.version_updated the latest Dag version is overwritten in place, having no task instances

The metric has not been released yet, so there is no deprecation path and no newsfragment.

The paths that write no serialized data — a bundle-metadata refresh, or a parse where nothing changed — are still uncounted.

related: #68906


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Co-authored-by: [Claude Opus 5] following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@SameerMesiah97 SameerMesiah97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good.

I think this does need a newsfragment as you are adding a new metric. At least, that was what a maintainer asked for when reviewing 2 recent PRs of mine that were adding new metrics to the Triggerer.

@Ei-Sandi
Ei-Sandi force-pushed the rename-dag-serialisation-metrics branch from 8d97d93 to b6f6473 Compare August 1, 2026 21:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants