GH-51046: [Python][Interchange] Preserve the categorical ordered flag in from_dataframe

[Kayvan-Zahiri]

Rationale for this change

from_dataframe builds the dictionary array without the is_ordered flag the producer reports, so an ordered categorical column comes back unordered. A pyarrow table with dictionary(int32, string, ordered=True) does not survive its own round trip, and an ordered pandas Categorical loses its ordering on the way in. The pandas consumer passes the flag through (pd.Categorical(values, categories=categories, ordered=categorical["is_ordered"])), and our own producer reports it, so pyarrow is the only side dropping it.

What changes are included in this PR?

categorical_column_to_dictionary passes ordered=categorical["is_ordered"] to DictionaryArray.from_arrays. Indices, dictionary and null handling are untouched.

Are these changes tested?

Yes. test_pyarrow_roundtrip_categorical is now parametrized over ordered True and False, and the ordered cases fail without the change: assert table.equals(result) sees ordered=1 going in and ordered=0 coming out. I also fixed a copy-paste in test_pandas_roundtrip_categorical, which read describe_categorical from the result column twice, so its is_ordered assertion compared the result with itself and could never fail.

Are there any user-facing changes?

An ordered dictionary column stays ordered through from_dataframe. No API change.

I used an AI assistant while finding and writing this. I checked the behavior and the tests myself before opening the PR.

• GitHub Issue: [Python][Interchange] from_dataframe drops the categorical ordered flag #51046

[github-actions]

Thanks for opening a pull request!

This pull request has been automatically converted to a draft because its title doesn't match Arrow's required format.

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

After updating the title, you can mark the pull request as ready for review.

See also:

• Other pull requests
• Contribution Guidelines - Contributing Overview

[github-actions]

⚠️ GitHub issue #51046 has been automatically assigned in GitHub to PR creator.

[jorisvandenbossche]

This looks like a good fix, but note that the interchange protocol is deprecated, and so IMO it is not worth it spending much time on it to update / fix things.

[Kayvan-Zahiri]

Understood, and that is your call to make. Happy to close this if you would rather not
carry it.

One thing worth separating out before I do, because it is independent of the interchange
question. The existing test_pandas_roundtrip_categorical compares the result to itself:

desc_cat_table = col_result.describe_categorical
desc_cat_result = col_result.describe_categorical

assert desc_cat_table["is_ordered"] == desc_cat_result["is_ordered"]
assert desc_cat_table["is_dictionary"] == desc_cat_result["is_dictionary"]

Both sides are col_result, so those two assertions hold for any input and the test cannot
fail on a round-trip mismatch. That is presumably why the dropped ordered flag went
unnoticed. The one-word change to col_table on the first line makes it an actual
comparison.

So there are two things in this PR: the ordered= argument in
categorical_column_to_dictionary, which sits squarely in the deprecated protocol, and a
test that silently passes, which does not.

Tell me which you prefer and I will do it:

• Close the whole thing, no hard feelings.
• Keep just the test fix, dropping the from_dataframe.py change, so the assertion is real
  for as long as the module is around.
• Merge as is, since it is +7/-4 with the parametrized case already written.

Not looking to spend your time on it either way. I raised it because a test that cannot
fail seemed worth mentioning regardless of what happens to the protocol.