ci: resolve Arrow and Parquet from conda-forge on Unix legs - #853
Open
abnobdoss wants to merge 1 commit into
Open
ci: resolve Arrow and Parquet from conda-forge on Unix legs#853abnobdoss wants to merge 1 commit into
abnobdoss wants to merge 1 commit into
Conversation
abnobdoss
marked this pull request as draft
July 26, 2026 19:44
abnobdoss
force-pushed
the
ci/add-conda-arrow
branch
from
July 28, 2026 01:13
c7093cd to
a1267a7
Compare
abnobdoss
marked this pull request as ready for review
July 28, 2026 02:15
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #799 (Cutting CI build times: prebuilt Arrow and build configuration). This is the conda-forge Arrow piece, Unix legs only.
What
Adds
.github/actions/setup-conda-arrow, a composite action that installslibarrowandlibparquetfrom conda-forge and exportsCMAKE_PREFIX_PATHand the loader path.Used on the Unix legs that build Arrow from source today: the
ubuntuandmacosjobs in Test, ASAN/UBSAN, SQL Catalog's non-Windows slices, and the AWS system-SDK slice.Why not the rest
Windows stays vendored because conda-forge only ships Release builds, so a Debug consumer would hit a CRT mismatch (
/MDdvs/MD). That is a packaging limitation rather than something to sequence around.Hive also stays vendored, since prebuilt Arrow exports no
thrift::thrifttarget andiceberg_hiverequires one. I plan to raise that as a follow-up PR.The AWS bundled-SDK slices cannot use conda at all. Bundling makes Arrow build the AWS SDK into its own
arrow_bundled_dependenciesarchive, and that only exists when Arrow is built from source, so only the system-SDK slice is converted. The Meson legs are untouched because they do not use Arrow.Notes
install-arrow.shreadsICEBERG_ARROW_BUILD_VERSIONfromIcebergThirdpartyToolchain.cmakeinstead of repeating the version, so bumping the vendored Arrow moves the conda legs with it. The pin is exact (libarrow==24.0.0), since conda treats=as a prefix match and patch releases do get published.ICEBERG_REQUIRE_SYSTEM_ARROW=ONtellsbuild_iceberg.shto fail the leg if Arrow was built from source anyway. That check is needed becausefetchcontent_declare(VendoredArrow ...)passesFIND_PACKAGE_ARGS: a failedfind_packagefalls back to a source build with no warning, and the leg would still pass.Validation
Converted legs go from 931 to 596 compile requests, and the rest are unchanged.
All 20 checks pass on the fork with the guard armed, so a green run does mean conda Arrow was resolved. Upstream reports the checks as skipped rather than passing, because every job is gated on
draft == false.