Conversation
Convert the last 7 macro fast-paths and 1 struct-field read in src/arrow/python/ to stable C-API equivalents so the translation units build under Py_LIMITED_API=0x030B0000: - common.h PyBytesView memoryview: PyMemoryView_GET_BUFFER (struct read) -> PyObject_GetBuffer/PyBuffer_Release, holding the contiguous memoryview via `ref` (fixes a use-after-free for non-contiguous memoryviews, where the prior ref was dropped without DECREF). - iterators.h / numpy_convert.cc: PySequence_ITEM / PySequence_Fast_GET_* -> PySequence_GetItem / PySequence_Fast + PyList_GetItem (uniform across GIL-enabled and GIL-disabled; PySequence_ITEM is a hidden struct-access macro unavailable under the limited API). - python_to_arrow.cc: PyTuple_GET_SIZE / PyList_GET_SIZE -> PyTuple_Size / PyList_Size. - common.cc: ty->tp_name struct read -> PyObject_StdStringTypeName. All introduced functions verified to resolve under Py_LIMITED_API=0x030B0000. Signed-off-by: Fathi Boudra <fathi.boudra@linaro.org>
PyGILState_Check() is a full-C-API function (declared only in cpython/pystate.h, absent from the public header) and is unavailable under Py_LIMITED_API=0x030B0000. In the non-freethreading cp311-abi3 build we ship, it reduces to "the current thread has a valid thread state and holds the GIL", which for any thread executing Python code is equivalent to Py_IsInitialized(). Py_IsInitialized() is limited-API-safe and still guards the post-finalization case (apacheGH-38626). It clears PyGILState_Check from all 13 translation units that pulled it in via common.h. Signed-off-by: Fathi Boudra <fathi.boudra@linaro.org>
These vendored pythoncapi_compat shims reference PyFrameObject, which is only defined when frameobject.h is pulled in, and an abi3 build (Py_LIMITED_API) does not include it. No arrow source calls the PyFrame* API, so guard the two frame blocks so the limited-API build compiles. Signed-off-by: Fathi Boudra <fathi.boudra@linaro.org>
Every pyarrow C++ translation unit compiles clean under Py_LIMITED_API=0x030B0000 (cp311-abi3). - datetime.h: define a layout-compatible PyDateTime_CAPI struct plus the PyDate_Check / PyDate_FromDate / PyTime_FromTime / PyDelta_FromDSU macros under Py_LIMITED_API. CPython's <datetime.h> is entirely absent in an abi3 build, but the datetime C-API is still exposed at runtime through the "datetime.datetime_CAPI" capsule that InitDatetime() already imports. - datetime.h/.cc, python_to_arrow.cc: read date/time/datetime/timedelta fields through the stable attribute API (PyDatetimeField) instead of the struct-field accessors hidden under Py_LIMITED_API; change the to_s/_ms/_us helpers to take PyObject*. - python_to_arrow.cc: replace PyList_GetItemRef/PyDict_GetItemStringRef-style 3.13+ calls with stable PyList_GetItem / PyDict_GetItem* equivalents. - extension_type.cc: replace PyWeakref_GetRef (3.12+) with the stable PyWeakref_GetObject. - helpers.cc: convert np.float16 via the __float__ protocol instead of the numpy scalar C-API (PyArray_IsScalar/PyArrayScalar_VAL, Half) which are hidden under Py_LIMITED_API; use PyTuple_New(0) instead of Py_GetConstantBorrowed(Py_CONSTANT_EMPTY_TUPLE). - numpy_to_arrow.cc: cast the opaque-under-limited-API numpy types to PyObject* at Python C-API boundaries; replace PyDict_GetItemStringRef with PyDict_GetItemString. Per-TU syntax audit: 23/23 translation units compile clean under Py_LIMITED_API=0x030B0000. Signed-off-by: Fathi Boudra <fathi.boudra@linaro.org>
fboudra
requested review from
AlenkaF,
pitrou,
raulcd and
rok
as code owners
September 17, 2026 10:24
|
Thanks for opening a pull request! This pull request has been automatically closed because you currently have 4 open pull requests, which is more than the limit of 3. Due to the increase in pull requests opened by AI bots, and in order to keep the review queue manageable, Apache Arrow limits contributors without repository access to at most 3 concurrently open pull requests. This helps make sure each pull request gets the attention it needs and that work in progress does not go stale. Once one of your other open pull requests has been merged or closed, you are welcome to reopen this one. See also: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
#50398
#50829
What changes are included in this PR?
Finishes the source-level conversion. After this PR every C++ translation unit in src/arrow/python/ compiles under
Py_LIMITED_API=0x030B0000, but the PR does not define the flag, so builds and CI are still on the full API.
The changes:
The last commit also fixes the remaining blockers in datetime, extension_type, numpy_to_arrow and python_to_arrow, which the per-TU syntax audit (23 translation units) found.
Are these changes tested?
Yes
Are there any user-facing changes?
No
Was AI used for this PR?
PR code and description written by:
Reviewed before submission by: