Skip to content

Standardize error handling in CLI connection import#64498

Open
Vegapunk-debug wants to merge 12 commits intoapache:mainfrom
Vegapunk-debug:main
Open

Standardize error handling in CLI connection import#64498
Vegapunk-debug wants to merge 12 commits intoapache:mainfrom
Vegapunk-debug:main

Conversation

@Vegapunk-debug
Copy link
Copy Markdown

@Vegapunk-debug Vegapunk-debug commented Mar 30, 2026

closes: None (No related issue)


Description

This PR standardizes error handling for the airflow connections import CLI operation, aligning its behavior with airflow variables import.

Note: This PR was originally opened to address transaction overhead by batching commits, but based on maintainer feedback (@potiuk), it was pivoted to prioritize robust error handling over performance.

Previously, a single database or validation error during a bulk connection import would cause the entire command to fail.

This PR updates connection_command.py by wrapping the individual session.merge() and session.commit() operations inside a try...except block within the _import_helper loop. If an individual connection fails to import, it now prints the specific error, executes a session.rollback(), and safely continues to the next connection in the file.

  • connection_command.py: Added per-item try/except block with session.rollback() to ensure single failures do not abort the entire batch import.
  • variable_command.py: Reverted previous optimization attempts to maintain its existing, intentional per-item commit and error handling structure.
Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in [airflow-core/newsfragments](https://github

@boring-cyborg
Copy link
Copy Markdown

boring-cyborg bot commented Mar 30, 2026

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

  • Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
  • Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

@potiuk
Copy link
Copy Markdown
Member

potiuk commented Mar 30, 2026

This one does not need newsfragment.

@Vegapunk-debug
Copy link
Copy Markdown
Author

@potiuk , I have completely removed the file now.

@Vegapunk-debug
Copy link
Copy Markdown
Author

Hi @potiuk , I resolved the issue you mentioned, Can you please review the PR.

Copy link
Copy Markdown
Member

@potiuk potiuk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually - some errors will ony be raised if we try to commit (and then individual variables are skipped), so I do not think that change is good -> if anything, connection import should be updated to also follow this pattern.

@potiuk
Copy link
Copy Markdown
Member

potiuk commented Apr 1, 2026

There are usually small number of connections or variables to import - so this optimisation is really worse than handling individual commit errors

@Vegapunk-debug Vegapunk-debug changed the title Improve CLI import command performance by avoiding N+1 commits Standardize error handling in CLI connection import Apr 1, 2026
@Vegapunk-debug
Copy link
Copy Markdown
Author

Hi @potiuk ,

Thank you for your guidance!

I have updated the pull request based on your feedback. I completely reverted the optimization attempt in variable_command.py. Instead, I've added a try...except block with session.rollback() in connection_command.py to ensure that it properly handles individual connection import failures, aligning it with the behavior of variable imports.

Please let me know if this looks good to go!

@Vegapunk-debug Vegapunk-debug requested a review from potiuk April 1, 2026 12:42
@potiuk
Copy link
Copy Markdown
Member

potiuk commented Apr 1, 2026

It's different than variables. Can you use the same approach as there, please.

@potiuk potiuk marked this pull request as draft April 1, 2026 18:56
@potiuk
Copy link
Copy Markdown
Member

potiuk commented Apr 1, 2026

Converted to draft for now.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the airflow connections import CLI flow to handle per-connection database/merge failures without aborting the entire import, aligning the behavior more closely with the variables import command.

Changes:

  • Wraps session.merge() + session.commit() in a per-item try/except, rolling back on failure and continuing to the next connection.
  • Adds failure messaging for individual connection import errors.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Copy link
Copy Markdown
Author

@Vegapunk-debug Vegapunk-debug left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One question regarding error handling:
variables_import uses print(... {e!r}), but this may expose sensitive information.

Would you prefer keeping it consistent with variables_import, or should we avoid printing the full exception here?

Copy link
Copy Markdown
Author

@Vegapunk-debug Vegapunk-debug left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@potiuk ,
Updated the implementation to follow variables_import approach.

@Vegapunk-debug Vegapunk-debug marked this pull request as ready for review April 2, 2026 04:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants