Pandas has deprecated certain aspects of .values (when it looses information), which our pandas->pyarrow conversion currently relies upon, so in the tests with pandas nightly we are seeing a lot of warnings because of that (https://github.com/ursacomputing/crossbow/actions/runs/34548523730/job/103106277981).
Small illustration:
>>> import pandas as pd
>>> import pyarrow as pa
>>> ser = pd.Series(pd.date_range("2025-01-01", periods=3, tz="UTC"))
>>> pa.array(ser)
<python-input-4>:1: Pandas4Warning: Series.values returning an ndarray that drops timezone information for DatetimeTZDtype is deprecated. In a future version, this will return the underlying DatetimeArray instead. Use 'Series.to_numpy()' to get a NumPy array, or 'Series.array' to get the ExtensionArray.
<pyarrow.lib.TimestampArray object at 0x7ff624fb1a20>
[
2025-01-01 00:00:00.000000Z,
2025-01-02 00:00:00.000000Z,
2025-01-03 00:00:00.000000Z
]
This is caused by our handling of a Series object in pa.array(..):
|
cdef object get_values(object obj, bint* is_series): |
|
if pandas_api.is_series(obj) or pandas_api.is_index(obj): |
|
result = pandas_api.get_values(obj) |
|
is_series[0] = True |
|
elif isinstance(obj, np.ndarray): |
|
result = obj |
|
is_series[0] = False |
|
else: |
|
result = pandas_api.series(obj, copy=False).values |
|
is_series[0] = False |
|
|
|
return result |
and
|
cpdef get_values(self, obj): |
|
""" |
|
Get the underlying array values of a pandas Series or Index in the |
|
format (np.ndarray or pandas ExtensionArray) as we need them. |
|
|
|
Assumes obj is a pandas Series or Index. |
|
""" |
|
self._check_import() |
|
if isinstance(obj.dtype, (self.pd.api.types.IntervalDtype, |
|
self.pd.api.types.PeriodDtype)): |
|
return obj.array |
|
return obj.values |
So we already have some custom handling for Period/Interval, but that will also be needed for DatetimeTZDtype
Pandas has deprecated certain aspects of
.values(when it looses information), which our pandas->pyarrow conversion currently relies upon, so in the tests with pandas nightly we are seeing a lot of warnings because of that (https://github.com/ursacomputing/crossbow/actions/runs/34548523730/job/103106277981).Small illustration:
This is caused by our handling of a Series object in
pa.array(..):arrow/python/pyarrow/array.pxi
Lines 5396 to 5407 in 07be48c
and
arrow/python/pyarrow/pandas-shim.pxi
Lines 231 to 242 in 07be48c
So we already have some custom handling for Period/Interval, but that will also be needed for DatetimeTZDtype