GH-49831: [Python] Withhold annotations from Python wheel until they are complete by rok · Pull Request #50168 · apache/arrow

rok · 2026-06-14T19:11:13Z

Rationale for this change

Since 24.0, pyarrow ships annotations and py.typed marker while the type stubs are still incomplete. This makes type checkers trust the partial stubs and report false errors in downstream code (e.g. Module has no attribute "all" for pyarrow.compute). Some type checkers also consume bundled .pyi files even without py.typed, so the stubs need to be withheld from wheels for now. See #49831.

What changes are included in this PR?

This temporarily omits both pyarrow/py.typed and the bundled pyarrow-stubs / .pyi files from built wheels until the stubs are complete. Wheel-content validation now asserts that neither py.typed nor .pyi files are present, and wheel build scripts no longer request stub docstring injection while stubs are not installed.

Are these changes tested?

Wheel-content validation has been updated to check the intended absence of py.typed and .pyi files.

Are there any user-facing changes?

Type checkers no longer pick up pyarrow's incomplete stubs from wheels.

GitHub Issue: [Python] type checkers broken for pyarrow.compute and other sub-modules since PyArrow 24.0+ #49831

github-actions · 2026-06-14T19:11:42Z

⚠️ GitHub issue #49831 has been automatically assigned in GitHub to PR creator.

Copilot

Pull request overview

This PR aims to prevent downstream type checkers from trusting PyArrow’s currently-incomplete bundled type stubs by withholding the PEP 561 py.typed marker from the built wheel (per GH-49831), while keeping stubs and py.typed in-tree for ongoing CI/type-checking work.

Changes:

Configure the wheel build to exclude pyarrow/py.typed via wheel.exclude.
Add explanatory comments documenting the rationale and removal TODO.

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

dangotbanned · 2026-06-15T08:37:15Z

+# TODO(GH-49831): Temporarily do not install pyarrow-stubs into wheels.
+# The stubs are incomplete, and some type checkers consume .pyi files even
+# without the py.typed marker. Re-enable this when the stubs are complete.
+#


You might wanna look into this repo as a follow up 🎁

https://github.com/astral-sh/docstring-adder/

Oooh, if this works we can get rid of quite some logic, thanks!

Ah great - hope so too!

If it doesn't, it could still be worth opening an issue since it's being actively developed - they might just not have coverage for cython yet

I tested it and didn't get any docstrings of cythonic symbols, so I opened an issue astral-sh/docstring-adder#184.

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

dangotbanned

Thanks @rok

AFAICT, this should unblock narwhals-dev/narwhals#3560

Probably best to have someone familiar with your build process look this over, but LGTM

rok · 2026-06-24T01:12:46Z

@dangotbanned @raulcd I think this is ready for review. Stubs remain in the repo, they are checked but they are not packed into the wheel.

dangotbanned

Thanks @rok, I still (#50168 (review)) approve 😉

raulcd · 2026-06-25T08:43:32Z

@github-actions crossbow submit wheelcp313 python-sdist

raulcd

This looks good to me and the wheel validations ensure type hint files are not included anymore.
I am triggering a CI job to ensure all wheel platforms are covered and sdist doesn't fail either. If CI jobs are green I think we can merge.
cc @jorisvandenbossche in case you want to do a quick test from pandas side

raulcd · 2026-06-25T08:40:01Z

 echo "=== (${PYTHON_VERSION}) Building wheel ==="
 export PYARROW_BUNDLE_ARROW_CPP=ON
-export PYARROW_REQUIRE_STUB_DOCSTRINGS=ON
+# TODO(GH-49831): Re-enable when pyarrow-stubs are shipped in wheels again.


minor nit, this will be done as part of:

[Python] Type checking support #32609

Should we reference the issue where this has to be done? might be easier when we are grepping/searching?

Good point, changed.

github-actions · 2026-06-25T08:46:00Z

Revision: 5483c2f

Submitted crossbow builds: ursacomputing/crossbow @ actions-0bfec31cfa

Task	Status
python-sdist
wheel-macos-monterey-cp313-cp313-amd64
wheel-macos-monterey-cp313-cp313-arm64
wheel-manylinux-2-28-cp313-cp313-amd64
wheel-manylinux-2-28-cp313-cp313-arm64
wheel-musllinux-1-2-cp313-cp313-amd64
wheel-musllinux-1-2-cp313-cp313-arm64
wheel-windows-cp313-cp313-amd64

conbench-apache-arrow · 2026-06-29T15:29:47Z

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 889141a.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details.

* Apply dependabot dependency updates Batches the applicable open dependabot bumps into one commit so that pyproject.toml and uv.lock stay consistent: - ruff 0.15.20 -> 0.16.1 (#1203 proposed 0.16.0; 0.16.1 is the latest and adds only bug fixes and preview-rule changes on top of it) - types-setuptools 83.0.0.20260706 -> 83.0.0.20260724 (#1202) - types-python-dateutil 2.9.0.20260518 -> 2.9.0.20260716 (#1201) - types-tqdm 4.68.0.20260608 -> 4.70.0.20260805 (#1199) - neo4j-viz 1.6.0 -> 1.7.0 (#1200, lock only; the >=1.6.0 floor stays put so the constraint remains permissive for library consumers) - aiohttp 3.14.1 -> 3.14.3 (#1214) - cryptography 49.0.0 -> 50.0.0 (#1216) Also re-resolves the whole lockfile with `uv lock --upgrade` rather than only the packages dependabot filed PRs for, which moves 48 packages in total. The ones worth noting beyond the list above: pyarrow 24.0.0 -> 25.0.0, tqdm 4.68.4 -> 4.70.0 and multimethod 2.0.2 -> 2.1 are runtime dependencies; pandas goes 3.0.3 -> 3.0.5; twine 6.2.0 -> 7.0.0 affects the publish group only; and sphinx-autodoc-typehints 2.3.0 -> 3.0.1 plus alabaster 0.7.16 -> 1.0.0 are major bumps but transitive only, since neither is in the sphinx extension list and the theme is the vendored neo4j one. pyarrow 25 needs the mypy.ini change. It withdraws the py.typed marker and inline annotations that 24 shipped (apache/arrow#50168), because those annotations were incomplete and broke type checkers on submodules such as pyarrow.compute (apache/arrow#49831). Without py.typed, strict mypy reports 26 import-untyped errors across the arrow client. Restoring ignore_missing_imports for pyarrow reverts exactly what #1164 removed when it adopted pyarrow 24, so this is the pre-24 behaviour rather than a new relaxation. pyarrow-stubs is not a usable alternative: it targets the pyarrow 20 API and yields 35 errors in 7 files. The stanza can go once upstream ships complete annotations (apache/arrow#32609). Verified: ruff 0.16.1 reformats nothing and reports no new findings (the 0.16 default-rule expansion does not reach us because tool.ruff.lint.select is explicit, and the new markdown formatting is excluded by include), mypy is clean, and the 632 unit tests pass. The sphinx API docs still build with -W and the rendered HTML is byte-identical to a pre-upgrade build, so the sphinx transitive majors change no output. nbconvert still produces byte-identical asciidoc for the tutorial notebooks despite the mistune 3.3.2 -> 3.3.4 bump. The torch 2.12.1 -> 2.13.0 bump (#1186, #1192) is still left out, for the same reason as in #1196 and re-checked today: the PyG wheel index for torch-2.13.0+cpu ships only pyg_lib, no torch-scatter or torch-sparse wheels, so notebook-ci would fall back to building those from source. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Drop older pyarrow version --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

Copilot AI review requested due to automatic review settings June 14, 2026 19:11

rok requested review from AlenkaF and raulcd as code owners June 14, 2026 19:11

Copilot started reviewing on behalf of rok June 14, 2026 19:11 View session

github-actions Bot added Component: Python awaiting committer review Awaiting committer review labels Jun 14, 2026

Copilot AI reviewed Jun 14, 2026

View reviewed changes

Comment thread python/pyproject.toml Outdated

rok mentioned this pull request Jun 14, 2026

[Python] type checkers broken for pyarrow.compute and other sub-modules since PyArrow 24.0+ #49831

Closed

rok force-pushed the broken-annotations-downstream branch from 325b6ac to 860479a Compare June 14, 2026 19:21

rok requested review from assignUser, jonkeane and kou as code owners June 14, 2026 19:21

github-actions Bot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Jun 14, 2026

dangotbanned reviewed Jun 14, 2026

View reviewed changes

Comment thread ci/scripts/python_wheel_validate_contents.py

Copilot AI review requested due to automatic review settings June 15, 2026 07:26

Copilot started reviewing on behalf of rok June 15, 2026 07:26 View session

github-actions Bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Jun 15, 2026

Copilot AI reviewed Jun 15, 2026

View reviewed changes

rok requested review from Copilot and dangotbanned June 15, 2026 08:03

Copilot started reviewing on behalf of rok June 15, 2026 08:10 View session

Copilot AI reviewed Jun 15, 2026

View reviewed changes

github-actions Bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Jun 15, 2026

dangotbanned approved these changes Jun 16, 2026

View reviewed changes

Comment thread python/CMakeLists.txt Outdated

rok force-pushed the broken-annotations-downstream branch from 1c4cec9 to fe246a4 Compare June 23, 2026 22:35

github-actions Bot removed the awaiting changes Awaiting changes label Jun 23, 2026

rok requested a review from dangotbanned June 24, 2026 10:36

dangotbanned approved these changes Jun 24, 2026

View reviewed changes

raulcd approved these changes Jun 25, 2026

View reviewed changes

github-actions Bot added awaiting merge Awaiting merge and removed awaiting change review Awaiting change review labels Jun 25, 2026

rok added 8 commits June 28, 2026 16:53

initial commit

699aff0

omit annotations from wheel

526998c

apacheGH-49831: Validate pyarrow stubs are omitted from wheels

9dfa372

apacheGH-49831: Keep wheel stub docstring counter commented

bc5f1b9

apacheGH-49831: Minimize temporary wheel validation changes

593e96b

review feedback

a4b6c4d

refactor

f1bc947

review feedback

9987aed

rok force-pushed the broken-annotations-downstream branch from 5483c2f to 9987aed Compare June 28, 2026 14:53

github-actions Bot added awaiting changes Awaiting changes and removed awaiting merge Awaiting merge labels Jun 28, 2026

rok mentioned this pull request Jun 28, 2026

[CI] Remove all __getattr__(name) -> Any from *.pyi files before next release #48970

Open

make sure no stub files are included

836cde2

github-actions Bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Jun 28, 2026

raulcd approved these changes Jun 29, 2026

View reviewed changes

raulcd merged commit 889141a into apache:main Jun 29, 2026
18 checks passed

raulcd removed the awaiting change review Awaiting change review label Jun 29, 2026

github-actions Bot added the awaiting merge Awaiting merge label Jun 29, 2026

FlorentinD mentioned this pull request Aug 6, 2026

Apply dependabot dependency updates neo4j/graph-data-science-client#1217

Merged

Conversation

rok commented Jun 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions Bot commented Jun 14, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

dangotbanned Jun 15, 2026

Choose a reason for hiding this comment

Uh oh!

rok Jun 15, 2026

Choose a reason for hiding this comment

Uh oh!

dangotbanned Jun 16, 2026

Choose a reason for hiding this comment

Uh oh!

rok Jun 23, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

dangotbanned left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rok commented Jun 24, 2026

Uh oh!

dangotbanned left a comment

Choose a reason for hiding this comment

Uh oh!

raulcd commented Jun 25, 2026

Uh oh!

raulcd left a comment

Choose a reason for hiding this comment

Uh oh!

raulcd Jun 25, 2026

Choose a reason for hiding this comment

Uh oh!

rok Jun 28, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jun 25, 2026

Uh oh!

Uh oh!

conbench-apache-arrow Bot commented Jun 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

rok commented Jun 14, 2026 •

edited

Loading