[Python] Improve the pyarrow build system

[jorisvandenbossche]

The current build system of pyarrow consists of using setuptools with a custom setup.py that calls cmake to do the actual build. While this works, there are nowadays newer python "build backends" that we could benefit from to make this easier to maintain and use.
Specifically build backends that are meant to work with CMake projects, such as scikit-build, might be worth exploring.

[raulcd]

Thanks @jorisvandenbossche !

[h-vetinari]

Specifically build backends that are meant to work with CMake projects, such as scikit-build, might be worth exploring.

Scipy has pioneered meson as the backend (in the wider python ecosystem at least), which is now being used by numpy & pandas as well, and is being picked up by more and more projects. For a regular python project it's probably a better choice than CMake-based builds (the syntax is already nice and pythonic too), though if you have substantial CMake infrastructure then that integrates more easily with scikit-build.

[jorisvandenbossche]

though if you have substantial CMake infrastructure then that integrates more easily with scikit-build.

We do have that, yes: https://github.com/apache/arrow/blob/main/python/CMakeLists.txt

[h-vetinari]

I'm aware, I've been building arrow for a while now. ;-)
It was more a rhetorical statement - depends a bit on what one considers "substantial" too. :)

[jorisvandenbossche]

Also relevant issue: #36410

[jorisvandenbossche]

Major wish-list item: proper incremental / cached cython builds.

[vyasr]

I'll note that if scikit-build is still of interest, the modern solution here would be to use scikit-build-core

[pitrou]

So, caching should be usage for us in the next release of Cython: cython/cython#6070 (comment)

[WillAyd]

We have recently been adding Meson configuration files to the Arrow repository, so as @h-vetinari suggests meson-python may be a good option here. In addition to SciPy, NumPy, and pandas using it, arrow-nanoarrow was ported to use it a few months back

[pitrou]

Well, the main concern here is really to get rid of our clumsy setup.py. We could also migrate our CMake setup to meson, but that seems to complicate things even more?

Also, how well-maintained and flexible are our Meson configuration files, compared to CMake? Which one is easier to work with?

[pitrou]

Is anyone ensuring that the Meson build files are up-to-date when the CMake files change? Are they compatible with anything else than gcc/clang?

We don't need two build systems in the source tree. I'm ok with migrating from CMake to Meson if desired (there has to be a convincing argument, though), I'm less ok with us having to care about two different build systems. Unless someone is specifically willing to maintain the Meson files so that others don't have to care.

[pitrou]

Oh, and one potentially important question: is Meson able to generate CMake configuration files for downstream projects? Given the prevalence of CMake as a build system for C++ projects, an Arrow C++ install should be discoverable using CMake.

[WillAyd]

Thanks for all of the questions.

Also, how well-maintained and flexible are our Meson configuration files, compared to CMake? Which one is easier to work with?

The Meson configuration files are pretty new, and need more development to be feature complete with CMake. Regarding the ease to work with, that's is of course subjective, but I would guess that the average Python developer would find Meson easier to work with than CMake, given the syntax similarity

Is anyone ensuring that the Meson build files are up-to-date when the CMake files change?

Not yet

Are they compatible with anything else than gcc/clang?

Yes Meson can be used by MSVC, Cygwin, and most other popular development tools that I know of. The main knock on the development requirements of Meson is that it requires Python. If that is or becomes a limiting factor for Arrow, there are alternative Meson implementations in other languages (muon is the most famous one I am aware of)

We don't need two build systems in the source tree.

I was (perhaps naively) thinking that this could be the status quo for a while. This is what we are doing in arrow-adbc and nanoarrow (granted those are much simpler projects).

Unless someone is specifically willing to maintain the Meson files so that others don't have to care.

I am happy to volunteer for this for the time being

is Meson able to generate CMake configuration files for downstream projects?

I am not aware of Meson being able to generate CMake configuration files. It looks like Meson projects could be included via ExternalProject but not FetchContent (at least according to this CMake discussion)

For transparency, I am not expert enough in CMake to enumerate all of the differences that would entail

an Arrow C++ install should be discoverable using CMake.

In addition to my comments above, Meson can easily produce pkg-config files for installation discovery

Hope that helps. @kou may be able to offer additional expertise on the topic, and @paleolimbot has a good deal of experience managing a Meson configuration side-by-side with CMake in arrow-nanoarrow

[kou]

is Meson able to generate CMake configuration files for downstream projects?

I am not aware of Meson being able to generate CMake configuration files. It looks like Meson projects could be included via ExternalProject but not FetchContent (at least according to this CMake discussion)

I think that this question means whether meson install can install XXXConfig.cmake or not (downstream projects can find a project by find_package(XXX) or not). I think that this question doesn't mean ExternalProject nor FetchContent.

Meson has a feature that generates XXXConfig.cmake: https://mesonbuild.com/CMake-module.html#cmakeconfigure_package_config_file

But it didn't work well when I tried it for Apache Arrow GLib. (It may work well now.)

an Arrow C++ install should be discoverable using CMake.

This discussion is only for PyArrow not Apache Arrow C++, right? If so, find_package(XXX) must not be matter. Because the current PyArrow CMake configuration doesn't provide CMake package for PyArrow.

I think that we can use either CMake or Meson for PyArrow. (We don't need to maintain both of them for PyArrow.)