Skip to content

Emit metric dag auto paused - #69078

Open
takayoshi-makabe wants to merge 3 commits into
apache:mainfrom
takayoshi-makabe:emit-metric-dag-auto-paused
Open

Emit metric dag auto paused#69078
takayoshi-makabe wants to merge 3 commits into
apache:mainfrom
takayoshi-makabe:emit-metric-dag-auto-paused

Conversation

@takayoshi-makabe

Copy link
Copy Markdown
Contributor

Emit dag.auto_paused metric when the scheduler automatically pauses a DAG after exceeding max_consecutive_failed_dag_runs. Platform teams can now alert on this event directly via StatsD/OpenTelemetry without polling the database or correlating failure metrics manually.

The metric includes dag_id and run_type tags via the existing stats_tags property on DagRun.

closes: #69004


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Claude Code


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

Comment thread airflow-core/src/airflow/models/dagrun.py
Comment thread airflow-core/tests/unit/models/test_dag.py
@henry3260

Copy link
Copy Markdown
Contributor

CI falling

@takayoshi-makabe

Copy link
Copy Markdown
Contributor Author

@henry3260

I fixed it.
da51ff1

The commands below execute successfully.

uv run python scripts/ci/prek/check_metrics_synced_with_the_registry.py airflow-core/src/airflow/models/dagrun.py

Could you please rerun the GitHub Actions?

@takayoshi-makabe

Copy link
Copy Markdown
Contributor Author

@henry3260
Sorry about that. The CI passed, but I updated the branch afterward. Could you please run it again?

@potiuk potiuk added the ready for maintainer review Set after triaging when all criteria pass. label Jul 2, 2026

@potiuk potiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks — useful metric. Auto-pausing after consecutive failures is exactly the kind of thing operators want alerting on, and today it happens silently. Emitting at the point of the pause in _check_last_n_dagruns_failed is the right place, and the test asserting the stats.incr call rather than just the pause behaviour is what makes it meaningful.

One inconsistency in the metric declaration inline.

Heads-up unrelated to the change itself: #70013 also edits metrics_template.yaml, in a nearby part of the file. Whichever of the two merges second will need a trivial rebase — no action now, just so it isn't a surprise.


Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting

legacy_name: "-"
name_variables: []

- name: "dag.auto_paused"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The declaration doesn't match what the code actually emits. stats.incr(..., tags=self.stats_tags) uses DagRun.stats_tags, which carries both dag_id and run_type — but this entry says name_variables: [] and the description mentions only "dag_id tagging".

For comparison, #70013 declares its tagged gauges as name_variables: ["dag_id"] with a matching legacy_name. Worth being consistent: either list both variables here with an appropriate legacy_name, or narrow the emission to {"dag_id": self.dag_id} if run_type isn't wanted on this counter. Given it counts a Dag-level event rather than a run-level one, dropping run_type may actually be the cleaner answer.

Also minor: the description's second line starts at the same indentation as the key, which makes it a slightly odd multi-line plain scalar. Indenting the continuation under description: reads better and matches the surrounding entries.


Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@potiuk

Thanks for the detailed review! I've dropped run_type from the tags so dag.auto_paused only carries dag_id (matching the "Dag-level event" reasoning you laid out), and fixed the description indentation to match the surrounding entries. CI is green now — happy to take another look whenever you get a chance.

commit: 98128f6


Drafted-by: Claude Code (Sonnet 5); reviewed by @takayoshi-makabe before posting

The metric was emitted in code but missing from the registry,
causing the check-metrics-synced-with-registry static check to fail.
@takayoshi-makabe
takayoshi-makabe force-pushed the emit-metric-dag-auto-paused branch from 732b68f to 98128f6 Compare July 30, 2026 13:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready for maintainer review Set after triaging when all criteria pass.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Emit metrics when Airflow auto pauses a DAG

3 participants