[C++][Python] Significant performance degradation after pyarrow 24.0.0 to 25.0.0 upgrade

[ghaarsma]

Describe the bug, including details regarding any error messages, version, and platform.

In our application we implement a local TimeSeries caching implementation that stores TimeSeries data into Parquet files. We heavily read/write parquet files via the pandas to_parquet(engine="pyarrow") and read_parquet. We also use the pyarrow.read_metadata directly from the pyarrows package. We use default (snappy) pandas compression when we write out our files, which consist of a single column DataFrame with the index being the timestamps (DateTimeIndex) and the single column representing the TimeSeries values (mostly Float32/64).

After installing pyarrow 25.0.0 we saw a significant reduction in performance (see trend below). When we downgraded back to 24.0.0 the problem went away. No other changes were made and we kept the rest of our dependencies the same.

The Dev/Prod stack Azure App Service on Linux 779de6246efc 6.6.139.1-1.azl3:

root@779de6246efc:/home/site/wwwroot# python
Python 3.13.13 (main, May  4 2026, 08:38:58) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyarrow as pa
>>> pa.show_info()
pyarrow version info
--------------------
Package kind              : python-wheel-manylinux228
Arrow C++ library version : 24.0.0
Arrow C++ compiler        : GNU 14.2.1
Arrow C++ compiler flags  :  -Wno-noexcept-type -Wno-self-move -Wno-subobject-linkage  -fdiagnostics-color=always  -Wall -fno-semantic-interposition -msse4.2
Arrow C++ git revision    :
Arrow C++ git description :
Arrow C++ build type      : release
PyArrow build type        : release

Platform:
  OS / Arch           : Linux x86_64
  SIMD Level          : avx2
  Detected SIMD Level : avx2

Memory:
  Default backend     : mimalloc
  Bytes allocated     : 0 bytes
  Max memory          : 0 bytes
  Supported Backends  : mimalloc, jemalloc, system

Optional modules:
  csv                 : Enabled
  cuda                : -
  dataset             : Enabled
  feather             : Enabled
  flight              : Enabled
  fs                  : Enabled
  gandiva             : -
  json                : Enabled
  orc                 : Enabled
  parquet             : Enabled
  opentelemetry       : Enabled

Filesystems:
  AzureFileSystem     : Enabled
  GcsFileSystem       : Enabled
  HadoopFileSystem    : Enabled
  S3FileSystem        : Enabled

Compression Codecs:
  brotli              : Enabled
  bz2                 : Enabled
  gzip                : Enabled
  lz4_frame           : Enabled
  lz4                 : Enabled
  snappy              : Enabled
  zstd                : Enabled

Component(s)

C++, Python

[raulcd]

Just to confirm you are using the provided wheels from PyPI, not building PyArrow yourselves, right?
From what I can see Py3.13 on Linux amd64 (wheel-manylinux-2-28-cp313).
Is there any possibility you could share a synthetic Parquet file where we could reproduce or a minimal reproducer? I don't think we've seen any other issue reporting a performance degradation so far, any help to reproduce would be appreciated.

[rok]

This is a large regression. Could you help narrow it down by:

• Trying pd.read_parquet(path, engine="pyarrow", pre_buffer=False) and pre_buffer=True, and reporting the timings? The ParquetFile default changed between 24 and 25, although pd.read_parquet ultimately uses pq.read_table, whose default was already True in Arrow 24.
• Reporting the pandas version and whether the DateTimeIndex is timezone-aware? Arrow 25 changed reconstructed named timezones from pytz to zoneinfo.ZoneInfo with pandas 3. If the cache compares timezone objects directly, cached ranges could be rejected even though the timestamps are equivalent. It would be useful to print type(df.index.tz), repr(df.index.tz), and the corresponding values for the requested timezone.
• Clarifying whether “Query Length (s)” is operation duration or the time range requested from the upstream API? If it is the latter, this may indicate changed cache-hit behavior rather than slower Parquet I/O.