Skip to content

Support a two-stage build for separate libimas-core / imas-core packaging#62

Open
SimonPinches wants to merge 1 commit into
iterorganization:developfrom
SimonPinches:feature/CMake
Open

Support a two-stage build for separate libimas-core / imas-core packaging#62
SimonPinches wants to merge 1 commit into
iterorganization:developfrom
SimonPinches:feature/CMake

Conversation

@SimonPinches
Copy link
Copy Markdown
Contributor

Add an opt-in AL_USE_INSTALLED_CORE option that lets the Python wrapper be built against a pre-installed al-core via find_package, instead of compiling and bundling libal alongside the Cython extensions. This enables splitting the project into two downstream packages (e.g. the libimas-core and imas-core conda packages) without forking the build system, while keeping the existing single-shot wheel build as the default.

  • New option AL_USE_INSTALLED_CORE (default OFF). When ON, the al library is located with find_package(al-core CONFIG REQUIRED) and aliased as al so the rest of the tree links transparently.
  • The C/C++ library block is gated with AND NOT AL_USE_INSTALLED_CORE so it is skipped in stage-2 builds.
  • Install an al-coreTargets export plus a generated al-coreConfig.cmake / al-coreConfigVersion.cmake (additive — also benefits any other downstream C++ consumer).
  • python/CMakeLists.txt early-returns when AL_USE_INSTALLED_CORE is ON, installing only the Cython extensions and skipping the libal / runtime-deps bundling that is not valid for an alias of an imported target.
  • Expose AL_USE_INSTALLED_CORE through pyproject.toml so the option flows through scikit-build-core via the matching env var.

Default behaviour is unchanged: with AL_USE_INSTALLED_CORE=OFF the build still compiles libal, builds the Python bindings, and produces the same self-contained wheel as before.

Inspired by conda-forge/staged-recipes#31554, but keeping CMake as the single source of truth rather than introducing a parallel Meson build.

…ging

Add an opt-in AL_USE_INSTALLED_CORE option that lets the Python wrapper
be built against a pre-installed al-core via find_package, instead of
compiling and bundling libal alongside the Cython extensions. This
enables splitting the project into two downstream packages (e.g. the
libimas-core and imas-core conda packages) without forking the build
system, while keeping the existing single-shot wheel build as the
default.

- New option AL_USE_INSTALLED_CORE (default OFF). When ON, the al
  library is located with find_package(al-core CONFIG REQUIRED) and
  aliased as `al` so the rest of the tree links transparently.
- The C/C++ library block is gated with `AND NOT AL_USE_INSTALLED_CORE`
  so it is skipped in stage-2 builds.
- Install an al-coreTargets export plus a generated al-coreConfig.cmake
  / al-coreConfigVersion.cmake (additive — also benefits any other
  downstream C++ consumer).
- python/CMakeLists.txt early-returns when AL_USE_INSTALLED_CORE is ON,
  installing only the Cython extensions and skipping the libal /
  runtime-deps bundling that is not valid for an alias of an imported
  target.
- Expose AL_USE_INSTALLED_CORE through pyproject.toml so the option
  flows through scikit-build-core via the matching env var.

Default behaviour is unchanged: with AL_USE_INSTALLED_CORE=OFF the
build still compiles libal, builds the Python bindings, and produces
the same self-contained wheel as before.

Inspired by conda-forge/staged-recipes#31554, but keeping CMake as the
single source of truth rather than introducing a parallel Meson build.
@SimonPinches
Copy link
Copy Markdown
Contributor Author

Windows CI fails due to stale vcpkg cache + flaky gitlab.dkrz.de mirror for libaec. Re-runing to try and succeed.

Cache restored from key: windows-2022-win-deps-a

The cache contains 52 of 58 vcpkg packages but is missing 6, including libaec, hdf5, and a couple of boost components. So every Windows job must re-download those six.

Five of the six come from GitHub and resolve in seconds. Only libaec is hosted on gitlab.dkrz.de, which is unreliable from GitHub-hosted Windows runners: │

Python 3.11 (passing): downloaded k202009-libaec-v1.1.6.tar.gz in ~2.4 s
Python 3.10 (failing): Attempt 1 → 22 s → timeout
Attempt 2 → 22 s → timeout
error: Download timed out (gave up at 45 s)

So gitlab.dkrz.de works for some jobs and not others within the same run — it's a pure upstream-flakiness coin-flip.

Why the cache never heals itself:

  • Failed to save: Unable to reserve cache with key windows-2022-win-deps-a, another job may be creating this cache.
  • GHA cache keys are immutable — whoever first saved under windows-2022-win-deps-a (very likely an earlier run that failed after installing 52 packages but before installing libaec) locked it in. Every subsequent save attempt — including from a fully-successful 58-package job — is rejected. So the cache is permanently stuck at 52 packages, forcing every build to roll the libaec dice.

Copy link
Copy Markdown
Collaborator

@olivhoenen olivhoenen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants