Skip to content

ci: resolve Arrow and Parquet from conda-forge on Unix legs - #853

Open
abnobdoss wants to merge 1 commit into
apache:mainfrom
abnobdoss:ci/add-conda-arrow
Open

ci: resolve Arrow and Parquet from conda-forge on Unix legs#853
abnobdoss wants to merge 1 commit into
apache:mainfrom
abnobdoss:ci/add-conda-arrow

Conversation

@abnobdoss

@abnobdoss abnobdoss commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

Part of #799 (Cutting CI build times: prebuilt Arrow and build configuration). This is the conda-forge Arrow piece, Unix legs only.

What

Adds .github/actions/setup-conda-arrow, a composite action that installs libarrow and libparquet from conda-forge and exports CMAKE_PREFIX_PATH and the loader path.

Used on the Unix legs that build Arrow from source today: the ubuntu and macos jobs in Test, ASAN/UBSAN, SQL Catalog's non-Windows slices, and the AWS system-SDK slice.

Why not the rest

Windows stays vendored because conda-forge only ships Release builds, so a Debug consumer would hit a CRT mismatch (/MDd vs /MD). That is a packaging limitation rather than something to sequence around.

Hive also stays vendored, since prebuilt Arrow exports no thrift::thrift target and iceberg_hive requires one. I plan to raise that as a follow-up PR.

The AWS bundled-SDK slices cannot use conda at all. Bundling makes Arrow build the AWS SDK into its own arrow_bundled_dependencies archive, and that only exists when Arrow is built from source, so only the system-SDK slice is converted. The Meson legs are untouched because they do not use Arrow.

Notes

install-arrow.sh reads ICEBERG_ARROW_BUILD_VERSION from IcebergThirdpartyToolchain.cmake instead of repeating the version, so bumping the vendored Arrow moves the conda legs with it. The pin is exact (libarrow==24.0.0), since conda treats = as a prefix match and patch releases do get published.

ICEBERG_REQUIRE_SYSTEM_ARROW=ON tells build_iceberg.sh to fail the leg if Arrow was built from source anyway. That check is needed because fetchcontent_declare(VendoredArrow ...) passes FIND_PACKAGE_ARGS: a failed find_package falls back to a source build with no warning, and the leg would still pass.

Validation

Converted legs go from 931 to 596 compile requests, and the rest are unchanged.

All 20 checks pass on the fork with the guard armed, so a green run does mean conda Arrow was resolved. Upstream reports the checks as skipped rather than passing, because every job is gated on draft == false.

@abnobdoss
abnobdoss marked this pull request as draft July 26, 2026 19:44
@abnobdoss
abnobdoss force-pushed the ci/add-conda-arrow branch from c7093cd to a1267a7 Compare July 28, 2026 01:13
@abnobdoss
abnobdoss marked this pull request as ready for review July 28, 2026 02:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant