Add support for pandas 3 based xcoms in airflow

[amoghrajesh]

---

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)

Supersedes #70558

Following discussion from Ash on https://lists.apache.org/thread/tbv4q04b3xx1hnmtq1b34tkzf43d5yqz

pandas 3 exposes its public classes from the pandas namespace, so a DataFrame is qualified as
pandas.DataFrame rather than pandas.core.frame.DataFrame. The serde registry is keyed on that
name and only knew the old one, so under pandas 3 no DataFrame could be pushed through XCom at
all:

TypeError: cannot serialize object of type <class 'pandas.DataFrame'>

Both names are now registered, so a DataFrame written by either pandas version can be read by
either. Verified across all four combinations (writer x reader), with the payload written by one
interpreter and read by the other:

Reader	pandas-2 payload	pandas-3 payload
pandas 2.3.3	object, missing = None	object, missing = None
pandas 3.0.5	str, missing = nan	str, missing = nan

The reader's pandas version decides the dtypes, not the writer's -- that, and the fact that a
component without this change cannot read a pandas-3-written DataFrame XCom, is what the
significant newsfragment documents.

Two provider tests also asserted on pandas 2 behaviour (object dtype, missing values stringified
to "nan" / "None") and are now version-aware. The production paths were already correct.

Reopened from #70558 -- carries the same three commits unchanged, plus one added test closing a
review gap: nothing previously forced deserialization through the registry entry for the other
pandas major's qualname, since serialize() only ever produces the qualname of whatever pandas is
actually installed. Depends on the separate revert of #70791 landing first.

related: #70558, #70791

Testing

Testing with pandas v2

DAG:

from __future__ import annotations

import pendulum

from airflow.sdk import DAG, task


@task
def push_df():
    import pandas as pd

    print(f"pandas version (push): {pd.__version__}")
    return pd.DataFrame({"strings": ["a", "b", None], "ints": [1, 2, None]})


@task
def pull_df(df):
    import pandas as pd

    print(f"pandas version (pull): {pd.__version__}")
    print(f"dtypes:\n{df.dtypes}")
    print(f"strings column: {df['strings'].tolist()}")
    print(f"ints column: {df['ints'].tolist()}")


with DAG(
    dag_id="pandas3_xcom_check",
    start_date=pendulum.datetime(2024, 1, 1, tz="UTC"),
    schedule=None,
    catchup=False,
):
    pull_df(push_df())

Testing with pandas v3

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

[amoghrajesh]

Ah I have to resolve conflicts

[amoghrajesh]

Compat failures are unrelated, likely fixed by #71145. I will review that

[amoghrajesh]

Failures fixed by #71145, merging this one.

[github-actions]

Backport failed to create: v3-3-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
❌	v3-3-test

You can attempt to backport this manually by running:

cherry_picker 4091ccf v3-3-test

This should apply the commit to the v3-3-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.