Skip to content

Limit pandas to < 3 for DataFrame XComs #70791

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
potiuk merged 8 commits into from
Aug 1, 2026
Merged

Limit pandas to < 3 for DataFrame XComs #70791

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
c2c0254
Limit pandas to < 3 for DataFrame XComs
potiuk Jul 30, 2026
2f55df0
Sync uv.lock with the pandas < 3 constraint
potiuk Jul 30, 2026
910d6ed
Add significant newsfragment for the pandas < 3 limit
potiuk Jul 30, 2026
50e577b
Apply the pandas < 3 limit to the remaining providers
potiuk Jul 30, 2026
693bd82
Merge branch 'main' into limit-pandas-below-3-for-dataframe-xcoms
vatsrahul1001 Jul 31, 2026
1e6dd6b
Regenerate provider docs index for the pandas < 3 pins
potiuk Jul 31, 2026
a56565f
Merge remote-tracking branch 'origin/limit-pandas-below-3-for-datafra…
potiuk Jul 31, 2026
3e611b2
Explain the pandas upper bound at every site and drop a vacuous test
potiuk Aug 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 35 additions & 0 deletions airflow-core/newsfragments/70791.significant.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
pandas is limited to ``< 3``; DataFrame XComs are refused on pandas 3

A ``pandas.DataFrame`` does not survive an XCom round trip under pandas 3 the way it does
under pandas 2, so this version of Airflow constrains pandas to ``< 3`` and refuses DataFrame
XComs at runtime when a newer pandas is installed anyway.

Two things change under pandas 3. Its public classes are exposed from the ``pandas``
namespace, so a DataFrame qualifies as ``pandas.DataFrame`` rather than
``pandas.core.frame.DataFrame`` and the XCom serializer registry -- keyed on the latter --
never dispatches to the pandas serializer. And the dtypes differ: an ``object`` column reads
back as ``str``, and missing values as ``nan`` rather than ``None``, so the *reader's* pandas
version decides what a task receives rather than the writer's.

Because pandas is an optional dependency, the ``< 3`` constraint alone cannot be relied on --
a deployment may install pandas 3 directly, or pull it in through another package. It is
therefore backed by a runtime check.

**Behaviour changes:**

- With pandas 3 or newer installed, pushing **or** pulling a ``DataFrame`` through XCom now
raises ``RuntimeError`` naming the installed pandas version and the supported range.
Reads are refused as well as writes, because a payload written under pandas 2 reads back
with pandas 3 dtypes and would otherwise hand the task different data than was pushed,
with no signal.
- Previously the same situation produced an opaque
``TypeError: cannot serialize object of type <class 'pandas.DataFrame'>`` from the
serializer registry, with nothing identifying pandas as the cause.
- Deployments that pass DataFrames through XCom must pin ``pandas < 3`` in the environment
that runs their tasks. Deployments that do not use DataFrame XComs are unaffected at
runtime.
- Nothing changes for pandas 2 users.

**This defers pandas 3 support rather than dropping it.** Moving from pandas 2 to pandas 3
is a deliberate migration for users to make, given the dtype changes it brings to data that
has already been written. Support for pandas 3 will be enabled in a future Airflow release.
8 changes: 4 additions & 4 deletions providers/amazon/docs/index.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -171,9 +171,9 @@ Install them when installing from PyPI. For example:
pip install apache-airflow-providers-amazon[aiobotocore]


==================== ============================================================================================================================================================
==================== =====================================================================================================================================================================
Extra Dependencies
==================== ============================================================================================================================================================
==================== =====================================================================================================================================================================
``aiobotocore`` ``aiobotocore>=3.0.0``
``cncf.kubernetes`` ``apache-airflow-providers-cncf-kubernetes>=7.2.0``
``s3fs`` ``s3fs>=2023.10.0``
Expand All @@ -186,14 +186,14 @@ Extra Dependencies
``imap`` ``apache-airflow-providers-imap``
``microsoft.azure`` ``apache-airflow-providers-microsoft-azure``
``mongo`` ``apache-airflow-providers-mongo``
``pandas`` ``pandas>=2.1.2; python_version <"3.13"``, ``pandas>=2.2.3; python_version >="3.13" and python_version <"3.14"``, ``pandas>=2.3.3; python_version >="3.14"``
``pandas`` ``pandas>=2.1.2,<3; python_version <"3.13"``, ``pandas>=2.2.3,<3; python_version >="3.13" and python_version <"3.14"``, ``pandas>=2.3.3,<3; python_version >="3.14"``
``openlineage`` ``apache-airflow-providers-openlineage>=2.3.0``
``salesforce`` ``apache-airflow-providers-salesforce``
``ssh`` ``apache-airflow-providers-ssh``
``standard`` ``apache-airflow-providers-standard``
``common.messaging`` ``apache-airflow-providers-common-messaging>=2.0.0``
``sqlalchemy`` ``sqlalchemy>=1.4.54``
==================== ============================================================================================================================================================
==================== =====================================================================================================================================================================

Downloading official packages
-----------------------------
Expand Down
15 changes: 12 additions & 3 deletions providers/amazon/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,9 +130,18 @@ dependencies = [
"apache-airflow-providers-mongo"
]
"pandas" = [
'pandas>=2.1.2; python_version <"3.13"',
'pandas>=2.2.3; python_version >="3.13" and python_version <"3.14"',
'pandas>=2.3.3; python_version >="3.14"',
# pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not
# round trip; capped until pandas 3 support lands.
# Tracked at https://github.com/apache/airflow/pull/70558
'pandas>=2.1.2,<3; python_version <"3.13"',
# pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not
# round trip; capped until pandas 3 support lands.
# Tracked at https://github.com/apache/airflow/pull/70558
'pandas>=2.2.3,<3; python_version >="3.13" and python_version <"3.14"',
# pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not
# round trip; capped until pandas 3 support lands.
# Tracked at https://github.com/apache/airflow/pull/70558
'pandas>=2.3.3,<3; python_version >="3.14"',
]
"openlineage" = [
"apache-airflow-providers-openlineage>=2.3.0"
Expand Down
12 changes: 6 additions & 6 deletions providers/apache/hdfs/README.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,19 +51,19 @@ The package supports the following python versions: 3.10,3.11,3.12,3.13,3.14
Requirements
------------

========================================== ==================================================================
========================================== ====================================================================
PIP package Version required
========================================== ==================================================================
========================================== ====================================================================
``apache-airflow`` ``>=2.11.0``
``apache-airflow-providers-common-compat`` ``>=1.12.0``
``hdfs[avro,dataframe,kerberos]`` ``>=2.5.4; python_version < "3.12"``
``hdfs[avro,dataframe,kerberos]`` ``>=2.7.3; python_version >= "3.12"``
``fastavro`` ``>=1.10.0; python_version >= "3.13" and python_version < "3.14"``
``fastavro`` ``>=1.12.1; python_version >= "3.14"``
``pandas`` ``>=2.1.2; python_version < "3.13"``
``pandas`` ``>=2.2.3; python_version >= "3.13" and python_version < "3.14"``
``pandas`` ``>=2.3.3; python_version >= "3.14"``
========================================== ==================================================================
``pandas`` ``>=2.1.2,<3; python_version < "3.13"``
``pandas`` ``>=2.2.3,<3; python_version >= "3.13" and python_version < "3.14"``
``pandas`` ``>=2.3.3,<3; python_version >= "3.14"``
========================================== ====================================================================

The changelog for the provider package can be found in the
`changelog <https://airflow.apache.org/docs/apache-airflow-providers-apache-hdfs/4.12.1/changelog.html>`_.
12 changes: 6 additions & 6 deletions providers/apache/hdfs/docs/index.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,19 +85,19 @@ Requirements

The minimum Apache Airflow version supported by this provider distribution is ``2.11.0``.

========================================== ==================================================================
========================================== ====================================================================
PIP package Version required
========================================== ==================================================================
========================================== ====================================================================
``apache-airflow`` ``>=2.11.0``
``apache-airflow-providers-common-compat`` ``>=1.12.0``
``hdfs[avro,dataframe,kerberos]`` ``>=2.5.4; python_version < "3.12"``
``hdfs[avro,dataframe,kerberos]`` ``>=2.7.3; python_version >= "3.12"``
``fastavro`` ``>=1.10.0; python_version >= "3.13" and python_version < "3.14"``
``fastavro`` ``>=1.12.1; python_version >= "3.14"``
``pandas`` ``>=2.1.2; python_version < "3.13"``
``pandas`` ``>=2.2.3; python_version >= "3.13" and python_version < "3.14"``
``pandas`` ``>=2.3.3; python_version >= "3.14"``
========================================== ==================================================================
``pandas`` ``>=2.1.2,<3; python_version < "3.13"``
``pandas`` ``>=2.2.3,<3; python_version >= "3.13" and python_version < "3.14"``
``pandas`` ``>=2.3.3,<3; python_version >= "3.14"``
========================================== ====================================================================

Downloading official packages
-----------------------------
Expand Down
15 changes: 12 additions & 3 deletions providers/apache/hdfs/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,9 +65,18 @@ dependencies = [
'hdfs[avro,dataframe,kerberos]>=2.7.3;python_version>="3.12"',
'fastavro>=1.10.0; python_version>="3.13" and python_version<"3.14"',
'fastavro>=1.12.1; python_version>="3.14"',
'pandas>=2.1.2; python_version <"3.13"',
'pandas>=2.2.3; python_version >="3.13" and python_version <"3.14"',
'pandas>=2.3.3; python_version >="3.14"',
# pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not
# round trip; capped until pandas 3 support lands.
# Tracked at https://github.com/apache/airflow/pull/70558
'pandas>=2.1.2,<3; python_version <"3.13"',
# pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not
# round trip; capped until pandas 3 support lands.
# Tracked at https://github.com/apache/airflow/pull/70558
'pandas>=2.2.3,<3; python_version >="3.13" and python_version <"3.14"',
# pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not
# round trip; capped until pandas 3 support lands.
# Tracked at https://github.com/apache/airflow/pull/70558
'pandas>=2.3.3,<3; python_version >="3.14"',
]

[dependency-groups]
Expand Down
12 changes: 6 additions & 6 deletions providers/apache/hive/README.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,19 +50,19 @@ The package supports the following python versions: 3.10,3.11,3.12,3.13,3.14
Requirements
------------

========================================== =================================================================
========================================== ====================================================================
PIP package Version required
========================================== =================================================================
========================================== ====================================================================
``apache-airflow`` ``>=2.11.0``
``apache-airflow-providers-common-compat`` ``>=1.12.0``
``apache-airflow-providers-common-sql`` ``>=1.32.0``
``hmsclient`` ``>=0.1.0``
``pandas`` ``>=2.1.2; python_version < "3.13"``
``pandas`` ``>=2.2.3; python_version >= "3.13" and python_version < "3.14"``
``pandas`` ``>=2.3.3; python_version >= "3.14"``
``pandas`` ``>=2.1.2,<3; python_version < "3.13"``
``pandas`` ``>=2.2.3,<3; python_version >= "3.13" and python_version < "3.14"``
``pandas`` ``>=2.3.3,<3; python_version >= "3.14"``
``pyhive[hive_pure_sasl]`` ``>=0.7.0``
``jmespath`` ``>=0.7.0``
========================================== =================================================================
========================================== ====================================================================

Optional cross provider package dependencies
--------------------------------------------
Expand Down
12 changes: 6 additions & 6 deletions providers/apache/hive/docs/index.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,19 +99,19 @@ Requirements

The minimum Apache Airflow version supported by this provider distribution is ``2.11.0``.

========================================== =================================================================
========================================== ====================================================================
PIP package Version required
========================================== =================================================================
========================================== ====================================================================
``apache-airflow`` ``>=2.11.0``
``apache-airflow-providers-common-compat`` ``>=1.12.0``
``apache-airflow-providers-common-sql`` ``>=1.32.0``
``hmsclient`` ``>=0.1.0``
``pandas`` ``>=2.1.2; python_version < "3.13"``
``pandas`` ``>=2.2.3; python_version >= "3.13" and python_version < "3.14"``
``pandas`` ``>=2.3.3; python_version >= "3.14"``
``pandas`` ``>=2.1.2,<3; python_version < "3.13"``
``pandas`` ``>=2.2.3,<3; python_version >= "3.13" and python_version < "3.14"``
``pandas`` ``>=2.3.3,<3; python_version >= "3.14"``
``pyhive[hive_pure_sasl]`` ``>=0.7.0``
``jmespath`` ``>=0.7.0``
========================================== =================================================================
========================================== ====================================================================

Optional cross provider package dependencies
--------------------------------------------
Expand Down
15 changes: 12 additions & 3 deletions providers/apache/hive/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,9 +63,18 @@ dependencies = [
"apache-airflow-providers-common-compat>=1.12.0",
"apache-airflow-providers-common-sql>=1.32.0",
"hmsclient>=0.1.0",
'pandas>=2.1.2; python_version <"3.13"',
'pandas>=2.2.3; python_version >="3.13" and python_version <"3.14"',
'pandas>=2.3.3; python_version >="3.14"',
# pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not
# round trip; capped until pandas 3 support lands.
# Tracked at https://github.com/apache/airflow/pull/70558
'pandas>=2.1.2,<3; python_version <"3.13"',
# pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not
# round trip; capped until pandas 3 support lands.
# Tracked at https://github.com/apache/airflow/pull/70558
'pandas>=2.2.3,<3; python_version >="3.13" and python_version <"3.14"',
# pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not
# round trip; capped until pandas 3 support lands.
# Tracked at https://github.com/apache/airflow/pull/70558
'pandas>=2.3.3,<3; python_version >="3.14"',
"pyhive[hive_pure_sasl]>=0.7.0",
"jmespath>=0.7.0",
]
Expand Down
8 changes: 4 additions & 4 deletions providers/common/sql/docs/index.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -141,18 +141,18 @@ Install them when installing from PyPI. For example:
pip install apache-airflow-providers-common-sql[pandas]


================== =======================================================================================================================================================================
================== ================================================================================================================================================================================
Extra Dependencies
================== =======================================================================================================================================================================
``pandas`` ``pandas[sql-other]>=2.1.2; python_version <"3.13"``, ``pandas>=2.2.3; python_version >="3.13" and python_version <"3.14"``, ``pandas>=2.3.3; python_version >="3.14"``
================== ================================================================================================================================================================================
``pandas`` ``pandas[sql-other]>=2.1.2,<3; python_version <"3.13"``, ``pandas>=2.2.3,<3; python_version >="3.13" and python_version <"3.14"``, ``pandas>=2.3.3,<3; python_version >="3.14"``
``openlineage`` ``apache-airflow-providers-openlineage``
``polars`` ``polars>=1.26.0``
``sqlalchemy`` ``sqlalchemy>=1.4.54``
``amazon`` ``apache-airflow-providers-amazon``
``datafusion`` ``datafusion>=50.0.0,<52.0.0``
``pyiceberg-core`` ``pyiceberg-core>=0.8.0``
``apache.iceberg`` ``apache-airflow-providers-apache-iceberg``
================== =======================================================================================================================================================================
================== ================================================================================================================================================================================

Downloading official packages
-----------------------------
Expand Down
15 changes: 12 additions & 3 deletions providers/common/sql/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,11 +72,20 @@ dependencies = [
# Any change in the dependencies is preserved when the file is regenerated
[project.optional-dependencies]
"pandas" = [
'pandas[sql-other]>=2.1.2; python_version <"3.13"',
# pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not
# round trip; capped until pandas 3 support lands.
# Tracked at https://github.com/apache/airflow/pull/70558
'pandas[sql-other]>=2.1.2,<3; python_version <"3.13"',
Comment thread
potiuk marked this conversation as resolved.
# Technically - we should add "sql-other" here as well, but this will only be possible when we move
# to sqlalchemy 2+ TODO(potiuk): do so.
'pandas>=2.2.3; python_version >="3.13" and python_version <"3.14"',
'pandas>=2.3.3; python_version >="3.14"',
# pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not
# round trip; capped until pandas 3 support lands.
# Tracked at https://github.com/apache/airflow/pull/70558
'pandas>=2.2.3,<3; python_version >="3.13" and python_version <"3.14"',
# pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not
# round trip; capped until pandas 3 support lands.
# Tracked at https://github.com/apache/airflow/pull/70558
'pandas>=2.3.3,<3; python_version >="3.14"',
]
"openlineage" = [
"apache-airflow-providers-openlineage"
Expand Down
Loading