Conversation
Convert the last 7 macro fast-paths and 1 struct-field read in src/arrow/python/ to stable C-API equivalents so the translation units build under Py_LIMITED_API=0x030B0000: - common.h PyBytesView memoryview: PyMemoryView_GET_BUFFER (struct read) -> PyObject_GetBuffer/PyBuffer_Release, holding the contiguous memoryview via `ref` (fixes a use-after-free for non-contiguous memoryviews, where the prior ref was dropped without DECREF). - iterators.h / numpy_convert.cc: PySequence_ITEM / PySequence_Fast_GET_* -> PySequence_GetItem / PySequence_Fast + PyList_GetItem (uniform across GIL-enabled and GIL-disabled; PySequence_ITEM is a hidden struct-access macro unavailable under the limited API). - python_to_arrow.cc: PyTuple_GET_SIZE / PyList_GET_SIZE -> PyTuple_Size / PyList_Size. - common.cc: ty->tp_name struct read -> PyObject_StdStringTypeName. All introduced functions verified to resolve under Py_LIMITED_API=0x030B0000. Signed-off-by: Fathi Boudra <fathi.boudra@linaro.org>
PyGILState_Check() is a full-C-API function (declared only in cpython/pystate.h, absent from the public header) and is unavailable under Py_LIMITED_API=0x030B0000. In the non-freethreading cp311-abi3 build we ship, it reduces to "the current thread has a valid thread state and holds the GIL", which for any thread executing Python code is equivalent to Py_IsInitialized(). Py_IsInitialized() is limited-API-safe and still guards the post-finalization case (apacheGH-38626). It clears PyGILState_Check from all 13 translation units that pulled it in via common.h. Signed-off-by: Fathi Boudra <fathi.boudra@linaro.org>
These vendored pythoncapi_compat shims reference PyFrameObject, which is only defined when frameobject.h is pulled in, and an abi3 build (Py_LIMITED_API) does not include it. No arrow source calls the PyFrame* API, so guard the two frame blocks so the limited-API build compiles. Signed-off-by: Fathi Boudra <fathi.boudra@linaro.org>
Every pyarrow C++ translation unit compiles clean under Py_LIMITED_API=0x030B0000 (cp311-abi3). - datetime.h: define a layout-compatible PyDateTime_CAPI struct plus the PyDate_Check / PyDate_FromDate / PyTime_FromTime / PyDelta_FromDSU macros under Py_LIMITED_API. CPython's <datetime.h> is entirely absent in an abi3 build, but the datetime C-API is still exposed at runtime through the "datetime.datetime_CAPI" capsule that InitDatetime() already imports. - datetime.h/.cc, python_to_arrow.cc: read date/time/datetime/timedelta fields through the stable attribute API (PyDatetimeField) instead of the struct-field accessors hidden under Py_LIMITED_API; change the to_s/_ms/_us helpers to take PyObject*. - python_to_arrow.cc: replace PyList_GetItemRef/PyDict_GetItemStringRef-style 3.13+ calls with stable PyList_GetItem / PyDict_GetItem* equivalents. - extension_type.cc: replace PyWeakref_GetRef (3.12+) with the stable PyWeakref_GetObject. - helpers.cc: convert np.float16 via the __float__ protocol instead of the numpy scalar C-API (PyArray_IsScalar/PyArrayScalar_VAL, Half) which are hidden under Py_LIMITED_API; use PyTuple_New(0) instead of Py_GetConstantBorrowed(Py_CONSTANT_EMPTY_TUPLE). - numpy_to_arrow.cc: cast the opaque-under-limited-API numpy types to PyObject* at Python C-API boundaries; replace PyDict_GetItemStringRef with PyDict_GetItemString. Per-TU syntax audit: 23/23 translation units compile clean under Py_LIMITED_API=0x030B0000. Signed-off-by: Fathi Boudra <fathi.boudra@linaro.org>
Add Py_LIMITED_API=0x030B0000 as a PUBLIC compile definition on the arrow_python target (inherited by the parquet_encryption and flight sub-targets) and pass --limited-api to Cython so the generated C++ is also limited-API. This is the single build path for the cp311-abi3 wheel covering all non-freethreading CPython 3.11-3.15. Signed-off-by: Fathi Boudra <fathi.boudra@linaro.org>
PyWeakref_GetObject (the cp311-abi3 replacement for PyWeakref_GetRef) returns the referent as a borrowed reference, or Py_None (not NULL) if the referent is dead. PyExtensionType::GetInstance returned the borrowed result as if it were a new reference, so the Python-side DECREF underflowed the T instance refcount, the non-deterministic heap corruption behind test_pandas' extension-type segfaults. GetInstance now INCREFs the alive result and treats Py_None as the dead case. Also carries the earlier limited-API defect fixes: - ABC bases removed from cdef classes (ABCMeta tp_new vs PyType_Spec heap types on CPython >= 3.14); membership done via mixin assignment and ABC.register at the Python layer - OwnedRef wrappers around borrowed PyList_GetItem / PySequence_Fast results removed (iterators.h, numpy_convert.cc, python_to_arrow.cc) - PySequence_Fast now returns exact tuples unchanged: dispatch on the result type (PyList_* vs PyTuple_*) - PyObject_GetBuffer in common.h passes PyBUF_READ | PyBUF_STRIDES and RETURN_IF_PYERROR Verified on CPython 3.13 and 3.14: test_pandas.py 384 passed, test_extension_type/test_types/test_array 507 passed, and the 200-iteration extension-type stress loop passes 200/200 on both. Signed-off-by: Fathi Boudra <fathi.boudra@linaro.org>
discover_tz_dir() now checks the TZDB environment variable before the hard-coded candidates and returns it when it names a directory, matching the upstream date library. This lets users point pyarrow at a pip tzdata zoneinfo directory on hosts whose system zoneinfo is partial (posix-only). Signed-off-by: Fathi Boudra <fathi.boudra@linaro.org>
Two local patches to the vendored header, which references private CPython internals that Py_LIMITED_API builds do not see: - Skip the PyUnicodeWriter* shims under Py_LIMITED_API: they reference _PyUnicodeWriter, while the real PyUnicodeWriter_* functions are stable since 3.3 and always available in limited-API builds. - PyLong_GetSign: derive the sign with PyObject_RichCompareBool against zero instead of _PyLong_Sign, and use %U for the type in the error message instead of reading tp_name. Signed-off-by: Fathi Boudra <fathi.boudra@linaro.org>
The test passed Py_None and then asserted on the raw Py_REFCNT value. For immortal objects, CPython >= 3.14 applies the saturating Py_INCREF while skipping the matching Py_DECREF, so a balanced incref/decref pair drifts the field by +1 per pair and the assertion is meaningless on Py_None. Pass a fresh empty list instead, which keeps the refcount assertion valid on every interpreter. Signed-off-by: Fathi Boudra <fathi.boudra@linaro.org>
libarrow_python referenced Py_REFCNT (stable-exported only since 3.14) and Cython's generated refcount macros (uniqueness check, dealloc keep-alive, immortal-table cleanup) did the same, so both libarrow_python.so and lib.abi3.so failed to import on 3.11-3.13. - python_test.cc: read the field directly via PyRawRefCnt (no header type layout is needed beyond the refcount slot, identical in 3.11-3.15). - Cython modules: add limited_api_compat.h that #defines Py_REFCNT to the raw field before codegen, and include it from common.pxd so the generated C++ sees the override after Python.h. Signed-off-by: Fathi Boudra <fathi.boudra@linaro.org>
DecimalFromString used PyObject_CallFunction with the 's#' format. The SizeT-remap of that call is compile-time-only: CPython 3.13+ headers (incl. the 3.14 headers this abi3 build compiles against) no longer redirect to _PyObject_CallFunction_SizeT, so the binary references the plain symbol, whose 3.11 implementation rejects '#' formats with a SystemError (check removed in 3.13). Build the string with PyUnicode_FromStringAndSize and call the constructor with PyObject_CallFunctionObjArgs instead. Both are stable limited-API and involve no format specifiers. Signed-off-by: Fathi Boudra <fathi.boudra@linaro.org>
Set SUFFIX to .abi3.so on the Cython module targets (POSIX; Windows keeps .pyd) so a single cp311-abi3 wheel is importable on every supported interpreter: importlib's EXTENSION_SUFFIXES includes '.abi3.so' on all of them. Also note in pyproject.toml that the cp311-abi3 tag comes from the SKBUILD_WHEEL__PY_API env override. Signed-off-by: Fathi Boudra <fathi.boudra@linaro.org>
Checks that every Py* symbol a built extension imports is exported by a reference CPython 3.11 (readelf on POSIX, objdump for Windows .pyd) and fails on any miss. Catches stable-ABI regressions such as symbols only exported since 3.13/3.14 (Py_REFCNT) before they ship in a cp311-abi3 wheel. Signed-off-by: Fathi Boudra <fathi.boudra@linaro.org>
fboudra
requested review from
AlenkaF,
pitrou,
raulcd and
rok
as code owners
September 17, 2026 10:24
|
Thanks for opening a pull request! This pull request has been automatically closed because you currently have 4 open pull requests, which is more than the limit of 3. Due to the increase in pull requests opened by AI bots, and in order to keep the review queue manageable, Apache Arrow limits contributors without repository access to at most 3 concurrently open pull requests. This helps make sure each pull request gets the attention it needs and that work in progress does not go stale. Once one of your other open pull requests has been merged or closed, you are welcome to reopen this one. See also: |
github-actions
Bot
added
Component: C++
Component: Python
awaiting review
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
#50398
#50829
What changes are included in this PR?
Adds python/scripts/audit_limited_api_symbols.py. It reads the dynamic symbol table of each built extension (a pure stdlib ELF reader on POSIX, objdump for Windows .pyd), collects every Py* symbol it imports, and checks that a reference CPython 3.11 exports it. Any miss fails the audit.
This catches stable ABI regressions, such as a symbol that is only exported since 3.13 or 3.14 (Py_REFCNT), before it ends up in a cp311-abi3 wheel. It is a standalone script in this PR; the following CI PR wires it into the wheel builds as a gate.
Are these changes tested?
Yes
Are there any user-facing changes?
No
Was AI used for this PR?
PR code and description written by:
Reviewed before submission by: