GH-50194: [C++] Move S3 and AWS-SDK to its own libarrow_s3.so

[raulcd]

Rationale for this change

Trying to reduce the size of libarrow.so and remove AWS SDK on some builds. Allow for users to plug and play based on requirements and divide our functionality into cleaner modules.

What changes are included in this PR?

Unconditionally build S3 and the AWS SDK into a different module libarrow_s3.so outside of libarrow.so.
Update bindings to link against the new libarrow_s3.so library.
Update the Linux Package jobs to have the new module into a different package.

Are these changes tested?

Yes via CI

Are there any user-facing changes?

Yes, users will need to either link against libarrow_s3.so or register using LoadFileSystemFactories

[raulcd]

@pitrou @kou I've been working on splitting the S3 library (and the AWS SDK) outside libarrow.so into its own library libarrow_s3.so.
On this PR I am just moving the AWS-SDK and the s3 filesystem related source into its own library. Any user trying to leverage it would require linking against it in order to use it (as we do with bindings) or dlopen via LoadFileSystemFactories (path), which would register at load time, no link dependency.
On this PR I am not planning on moving our existing bindings to the FileSystemFromUriAndOptions and LoadFileSystemFactories path. I am also not sure we should do that. I think that path is good for a user that doesn't want to link against libarrow_s3.so and have the same functionality but probably not what the majority of users (and our internal bindings) should do? What are your thoughts on that?
As per the size of the artifacts with the new code the size of libarrow.so and libarrow_s3.so:

$ ls -lhL libarrow.so libarrow_s3.so
-rwxrwxr-x 1 raulcd raulcd  59M Jun 22 19:30 libarrow_s3.so
-rwxrwxr-x 1 raulcd raulcd 317M Jun 22 19:29 libarrow.so

And we can see AWS symbols aren't present on libarrow.so

$  nm -C libarrow.so | grep -c "Aws::"
0
$ nm -C libarrow_s3.so | grep -c "Aws::"
33991

With current main libarrow.so size and it contains AWS SDK symbols:

$ ls -lhL libarrow.so
-rwxrwxr-x 1 raulcd raulcd 368M Jun 22 19:45 libarrow.so
$ ls -lhL libarrow_s3.so
ls: cannot access 'libarrow_s3.so': No such file or directory
$ nm -C libarrow.so | grep -c "Aws::"
33991

Those are debug builds but as a summary:
libarrow.so goes from 368M to 317M (~51M smaller), and AWS (33,991 symbols) move entirely into the new 59M libarrow_s3.so.

[kou]

I think that we should use LoadFileSystemFactories() for bindings to avoid loading the S3 module for users who don't need S3. For example, PyArrow users who also want to use the S3 module, they will install pyarrow_s3 (or something) explicitly.

I think that bindings can provide convenient API to use the S3 module even if we use LoadFileSystemFactories().

[raulcd]

I think that we should use LoadFileSystemFactories() for bindings to avoid loading the S3 module for users who don't need S3. For example, PyArrow users who also want to use the S3 module, they will install pyarrow_s3 (or something)` explicitly.

With conda this isn't necessary, we already ship all the .so as different packages allowing users to pick and choose what to install. This won't change, if a user installs pyarrow-core and installs libarrow-s3 will have S3 capabilities, same as if we install today pyarrow-core with libarrow-flight. If libarrow-s3 is not installed PyArrow would just ImportError when not finding the corresponding DLL. Basically we build with all capabilities turned on but install only the necessary .so and if not found they ImportError . I'll validate pyarrow fails with ImportError if the .so isn't present but this should behave as the other modules.

With wheels this is another different beast and I have to explore a little further. A related issue:

• [Python] usability improvements for a "minimal" pyarrow #38536

The original problem we had with wheels is that there's no mechanism to share dependencies between wheels. Auditwheel/delvewheel/delocate mangle the .so name to avoid other wheels clashing with other dependencies symbols. The problem is that libarrow_s3.so requires libarrow symbols and it's not clear how this pyarrow_s3 would be shipped. Should it include its own libarrow using the mangled symbols for the new wheel? Should it use the libarrow library coming from the main pyarrow wheel? What happens with different versions of pyarrow and pyarrow_s3 installed?

As a note, I've just validated we don't mangle libarrow (or any of our .so) on the wheels. I am going to start exploring this a little further to see if I can come up with something even though I am still unclear about some of the questions above, like version matching to avoid ABI problems.

Related: @amol- who worked on consolidatewheels in the past:

• https://github.com/amol-/consolidatewheels

And some Python PEP attempts to define some external dependencies for wheels are on discussion:
https://discuss.python.org/t/pep-725-specifying-external-dependencies-in-pyproject-toml-round-2/103890

What I am saying is that using LoadFileSystemFactories() isn't solving the real problem which in my opinion is: how do we share a single libarrow between several extra wheels and coordinate versioning?

cc @h-vetinari who knows this space and might shed some light

[raulcd]

cc @jorisvandenbossche

[h-vetinari]

For the PyPI side, you might be able to do something similar to what numpy/scipy are doing with openblas as a wheel.

[pitrou]

The AND WIN32 isn't useful, right?

[raulcd]

We use the same pattern on other places:

  if(ARROW_BUILD_STATIC AND WIN32)
    target_compile_definitions(arrow_compute_static PUBLIC ARROW_COMPUTE_STATIC)
  endif()

or

if(ARROW_BUILD_STATIC AND WIN32)
  target_compile_definitions(arrow_static PUBLIC ARROW_STATIC)
endif()

Taking a look at the definition on visibility.h of ARROW_S3_STATIC is already guarded for WIN32:

#if defined(_WIN32) || defined(__CYGWIN__)

So it will only be used on WIN32, it does not seem necessary on others so I would say the AND WIN32 does nothing but it's hygiene?

[pitrou]

So, this is as if ARROW_S3_MODULE was always enabled, right?

[raulcd]

So, this is as if ARROW_S3_MODULE was always enabled, right?

Yes but with a small caveat. ARROW_S3_MODULE bundled both into libarrow and generated a separate .so. This removes ARROW_S3_MODULE and makes ARROW_S3 unconditionally generate a new libarrow_s3.so and remove AWS SDK and s3fs.cc from libarrow.so. The bindings link against this new library. I have also validated the size reduction on libarrow.so and that no symbols for AWS are included in it.

[pitrou]

Oh, great, thank you!

[github-actions]

Revision: 55a438d

Submitted crossbow builds: ursacomputing/crossbow @ actions-db96d2f799

Task	Status
test-conda-cpp

[Copilot]

🔵 Needs a closer look

It changes cross-platform C++ linkage/visibility and multiple packaging/binding integration points, so a human should validate downstream build/runtime behavior across supported platforms.

Review details

• Files reviewed: 22/22 changed files
• Comments generated: 0 new
• Review effort level: Lite

[tadeja]

uf, grande changes, thanks @raulcd!
Here some hopefully useful pointers from my end using the last CI log Docker Test conda-cpp

at line 784

-- Providing CMake module for FindAWSSDKAlt as part of Arrow CMake package

-> A) It looks like AWS dependency remains with core Arrow instead of "ArrowS3 CMake package" during configure there?

at 2454

-- Installing: /opt/conda/envs/arrow/lib/cmake/Arrow/FindAWSSDKAlt.cmake

-> B) Here FindAWSSDKAlt.cmake is installed into cmake/Arrow - Doesn't it need to move to cmake/ArrowS3 ?

at 2472

-- Installing: /opt/conda/envs/arrow/lib/cmake/ArrowS3/ArrowS3Config.cmake
-- Installing: /opt/conda/envs/arrow/lib/cmake/ArrowS3/ArrowS3ConfigVersion.cmake
-- Installing: /opt/conda/envs/arrow/lib/cmake/ArrowS3/ArrowS3Targets.cmake

C) One more issue - installed ArrowS3Targets.cmake defines targets with no INTERFACE_LINK_LIBRARIES
Can be run in conda-cpp container after cpp_build.sh:

grep -c INTERFACE_LINK_LIBRARIES /opt/conda/envs/arrow/lib/cmake/ArrowS3/ArrowS3Targets.cmake

0
That isn't visible in current CI jobs, but could be verified in CI with the following additions, after discussions with 🤖 Fable, and committed here da9f947 ... See if it is useful for you to add changes along those lines to test-conda-cpp to see the failure here on PR too ?

[raulcd]

@github-actions crossbow submit test-conda-cpp

[Copilot]

🔵 Needs a closer look

It changes core C++ build/link/export semantics and downstream packaging/bindings across multiple ecosystems, so it needs a final human validation pass even after addressing the Meson issue.

Review details

• Files reviewed: 22/22 changed files
• Comments generated: 1
• Review effort level: Lite

[github-actions]

Revision: 142e154

Submitted crossbow builds: ursacomputing/crossbow @ actions-0ed0055ecc

Task	Status
test-conda-cpp

[raulcd]

0 That isn't visible in current CI jobs, but could be verified in CI with the following additions, after discussions with 🤖 Fable, and committed here da9f947 ... See if it is useful for you to add changes along those lines to test-conda-cpp to see the failure here on PR too ?

Maybe we could try to expand the following to cover static builds and other Arrow modules? But that's probably better done as a standalone issue / PR:

arrow/ci/scripts/cpp_test.sh

Lines 120 to 156 in a91ae52

	# This is for testing find_package(Arrow).
	#
	# Note that this is not a perfect solution. We should improve this
	# later.
	#
	# * This is ad-hoc
	# * This doesn't test other CMake packages such as ArrowDataset
	if [ "${ARROW_USE_MESON:-OFF}" = "OFF" ] && \
	[ "${ARROW_EMSCRIPTEN:-OFF}" = "OFF" ] && \
	[ "${ARROW_USE_ASAN:-OFF}" = "OFF" ] && \
	[ "${ARROW_USE_TSAN:-OFF}" = "OFF" ] && \
	[ "${ARROW_CSV:-ON}" = "ON" ]; then
	CMAKE_PREFIX_PATH="${CMAKE_INSTALL_PREFIX:-${ARROW_HOME}}"
	case "$(uname)" in
	MINGW*)
	# <prefix>/lib/cmake/ isn't searched on Windows.
	#
	# See also:
	# https://cmake.org/cmake/help/latest/command/find_package.html#config-mode-search-procedure
	CMAKE_PREFIX_PATH+="/lib/cmake/"
	;;
	esac
	if [ -n "${VCPKG_ROOT}" ] && [ -n "${VCPKG_DEFAULT_TRIPLET}" ]; then
	# Search vcpkg before <prefix>/lib/cmake.
	CMAKE_PREFIX_PATH="${VCPKG_ROOT}/installed/${VCPKG_DEFAULT_TRIPLET};${CMAKE_PREFIX_PATH}"
	fi
	cmake \
	-S "${source_dir}/examples/minimal_build" \
	-B "${build_dir}/examples/minimal_build" \
	-DCMAKE_PREFIX_PATH="${CMAKE_PREFIX_PATH}"
	cmake --build "${build_dir}/examples/minimal_build"
	pushd "${source_dir}/examples/minimal_build"
	# PATH= is for Windows.
	PATH="${CMAKE_INSTALL_PREFIX:-${ARROW_HOME}}/bin:${PATH}" \
	"${build_dir}/examples/minimal_build/arrow-example"
	popd
	fi

[raulcd]

Trying to test this it seems we might be missing Azure SDK libraries as a dependency for Arrow::arrow_static so find_package(Arrow) fails to allow consumer linking raising undefined Azure:: symbols. ARROW_GCS appends google-cloud-cpp::storage to ARROW_STATIC_INSTALL_INTERFACE_LIBS but there is no equivalent block for ARROW_AZURE.
We should probably increase coverage for testing those. I'll open a separate issue.

[tadeja]

Maybe we could try to expand the following to cover static builds and other Arrow modules? But that's probably better done as a standalone issue / PR:

arrow/ci/scripts/cpp_test.sh

Lines 120 to 156 in a91ae52

O, that's a much better location indeed! If it's not too late, here's a draft PR on cpp/examples/minimal_build/example.cc

I'd try adding Azure:: fix there too.

[tadeja]

Should this remaining ARROW_S3 section get an update now? I've tried it locally on top of PR changes, seems OK;

diff --git a/cpp/src/arrow/CMakeLists.txt b/cpp/src/arrow/CMakeLists.txt
@@ -111,22 +111,6 @@ if(ARROW_USE_GLOG)
   endif()
 endif()
 
-if(ARROW_S3)
-  if(AWSSDK_SOURCE STREQUAL "SYSTEM")
-    list(APPEND
-         ARROW_STATIC_INSTALL_INTERFACE_LIBS
-         aws-cpp-sdk-identity-management
-         aws-cpp-sdk-sts
-         aws-cpp-sdk-cognito-identity
-         aws-cpp-sdk-s3
-         aws-cpp-sdk-core)
-  elseif(AWSSDK_SOURCE STREQUAL "BUNDLED")
-    if(UNIX)
-      list(APPEND ARROW_STATIC_INSTALL_INTERFACE_LIBS CURL::libcurl)
-    endif()
-  endif()
-endif()
 if(ARROW_WITH_OPENTELEMETRY)

@@ -1070,6 +1054,10 @@ if(ARROW_FILESYSTEM)
     if(AWSSDK_SOURCE STREQUAL "SYSTEM")
       list(APPEND ARROW_S3_STATIC_INSTALL_INTERFACE_LIBS ${AWSSDK_LINK_LIBRARIES})
+    elseif(AWSSDK_SOURCE STREQUAL "BUNDLED")
+      if(UNIX)
+        list(APPEND ARROW_S3_STATIC_INSTALL_INTERFACE_LIBS CURL::libcurl)
+      endif()
     endif()
     add_arrow_lib(arrow_s3

[Copilot]

🔵 Needs a closer look

It changes core C++ library partitioning and downstream packaging/linkage across multiple build systems and languages, which warrants final human validation for ABI/packaging compatibility.

Review details

• Files reviewed: 22/22 changed files
• Comments generated: 0 new
• Review effort level: Lite

[tadeja]

(Sorry for CI re-run and void comments, I added my commits from #51280 onto branch GH-50194 here by accident. Those should be removed now.)