Repository navigation
Simplify OpenMP support setup #493
Description
Activity
@eddelbuettel Short version:
We still need to incorporate the
configure.acOpenMP test as Apple's Clang does not ship with OpenMP without explicit user interaction to enable it.Lines 118 to 123 in 554e7c1
Perhaps something similar to
data.table's new Clang 17 detection scheme for libomp clang 17 mismatch could be used to further refine it:Though, CRAN's macOS base R version will need to ship a newer OpenMP
libomp.dylibbefore pushing this further as a majority of the base is likely moving to macOS 26 (Tahoe).
The longer version:
-
OpenMP headers are still opt-in for local development and compilation, that is the following is still required:
Lines 20 to 23 in 554e7c1
The exception would be if we test for the presence of the runtime in
/usr/local/liband headers to/usr/local/includejust for macOS. Otherwise, Ripley's old check will bite. -
Headers for the opt-in would need to be downloaded and installed based on matching Apple Clang versions to the libomp version shipped with llvm as discussed here https://mac.r-project.org/openmp/
- You can get by with using either
macrtools::openmp_install()oropenmp-install.shshell script to handle the correct reference checks. - For an overview, see the OpenMP macOS shell project overview
- You can get by with using either
-
What's more interesting is CRAN's R macOS binaries ship with
libomp.dylibfound at$R_HOME/lib. (This corresponds to the Xcode version used on CRAN.)- This means that packages downloaded from CRAN are OpenMP enabled by default assuming no further compilation locally is required.
-
Presently, the runtimes diverge for compiling locally when special operations need to be performed (discussions in: fix installation using clang-17 Rdatatable/data.table#7318 (comment))
- The macOS CRAN build server is using clang 14.0.0. The current Xcode version on macOS 26: Tahoe is clang 17.0.0.
- Thus, we're seeing the error under certain OpenMP features like
schedule(dynamic)of:
symbol not found in flat namespace '___kmpc_dispatch_deinit'
Does this hopefully clarify the situation?
-
Does this hopefully clarify the situation?
Yes, thanks. My initial sketch here had retained the compilation test. I will retain the 'apple llvm' too. I may try to commit this to a branch. (I was not yet going for the
inlinepart.)I opened branch feature/openmp and added a quick
./configure; cat src/Makevarsto the ci script. On macos we get## -*- mode: makefile; -*- PKG_CPPFLAGS = -I../inst/include -DARMA_USE_CURRENT PKG_CXXFLAGS = PKG_LIBS= $(LAPACK_LIBS) $(BLAS_LIBS) $(FLIBS)
where as on leeenucks we get
## -*- mode: makefile; -*- PKG_CPPFLAGS = -I../inst/include -DARMA_USE_CURRENT PKG_CXXFLAGS = $(SHLIB_OPENMP_CXXFLAGS) PKG_LIBS= $(SHLIB_OPENMP_CXXFLAGS) $(LAPACK_LIBS) $(BLAS_LIBS) $(FLIBS)
Methinks we could simplify and always add
$(SHLIB_OPENMP_CXXFLAGS)because R takes care of keeping it empty when it needs to? But I may underestimate some specific macos situations. Theconfigure.acnow reflects what you posted last night, take a look (and please ignore the still-remaining, commented-out old bits).So, if we just relied upon:
PKG_CXXFLAGS = $(SHLIB_OPENMP_CXXFLAGS) PKG_LIBS = $(SHLIB_OPENMP_CXXFLAGS) $(LAPACK_LIBS) $(BLAS_LIBS) $(FLIBS)
How would
$(SHLIB_OPENMP_CXXFLAGS)propagate to#define ARMA_USE_OPENMP 1? I suppose that's the main driving factor as to why we still need some kind ofconfigure.ac.Glancing at: https://github.com/RcppCore/RcppArmadillo/tree/feature/openmp
This will be fine as it again keeps the status quo of forcing
#define ARMA_DONT_USE_OPENMP 1present in an Apple Clang world. If we move away from that, I'm worried downstream developers would be forced to opt-in to Simon's hack or we'd need to direct folks to use homebrew for clang 18 or clang 16 with libomp.If we move away from that
No my plan was to keep that in order to
- pass one control value on to Armadillo via `ARMA_USE_OPENMP
- re-use R's own knowledge via
$(SHLIB_OPENMP_CXXFLAGS)which "autofills"
This should work and I can test the Linux case via reverse dependencies. I can't really test the macOS case. (Well, I guess I could go to town and set up something new on GHA but I may not have quite the appetite for it outside of a possible one-off.)
PS In the 'can use OpenMP' \times 'are we on macOS' it seems I am still missing a cell in the 2 x 2 matrix. But I guess the (rarer ?) case of successsfully locally tweaked macOS with OpenMP would be found from the compilation test?
OpenMP × macOS
macOS Not macOS Can use OpenMP Manually configured Standard (Linux/Windows) Cannot use OpenMP Default macOS Rare/unsupported So we have four scenarios:
Non-macOS systems (Linux/Windows): OpenMP typically works out of the box.
SHLIB_OPENMP_CXXFLAGSis set correctly, compilation test passes, everything works as expected. There might be rare cases without OpenMP support, but the compilation test would catch those.Default macOS (no OpenMP installed): This is probably the most common macOS case. Since Apple doesn't ship OpenMP with Xcode, the compilation test fails and we correctly skip OpenMP features. No surprises here.
The interesting case - macOS with manually installed OpenMP: This is what you're asking about, right? If a developer has gone through the setup process (installing headers to
/usr/local/includeand runtime to/usr/local/lib), then yes, a configure-time compilation test should successfully detect it. The test would compile a simple OpenMP program, link against-lomp, and pass if everything's in place.The compilation test effectively identifies these "locally tweaked" macOS setups and enables OpenMP for them. So in that sense, it handles this case well.
The wrinkle is that even when the test passes on macOS, we might still hit the runtime version mismatch I mentioned earlier (Clang 14 on CRAN vs Clang 17 locally that has differing dylib's for OpenMP). The compilation test tells us OpenMP is available, but not whether specific features like
schedule(dynamic)will work without additional version checks.So I think the compilation test does solve the "is OpenMP present" question across all four scenarios. The version mismatch is a separate layer that might need additional handling for certain OpenMP features at least with this version of R on macOS with the current Xcode toolchain.
Does that make sense? Or, am I missing something about what you're envisioning?
Thanks for spelling out the two-by-two setup, and its four cases. I agree that we seem to be good in the three main ones (both 'not macos' and macos-without).
The question I have (as a non-macOS user) is whether the 'locally modified macos with openmp' is good enough. If the compiler and linker flags are passed to the configure script, then the simple compilation test should succeed, and we should be working there as well. Correct?
@eddelbuettel Almost, but there's a catch with how RcppArmadillo currently works.
The compilation test would succeed for locally modified macOS setups, you're right about that. But we currently hard-code the
ARMA_USE_OPENMPdefine into the distributed headers based on CRAN's build environment (configure.ac lines 95-114, RcppArmadilloConfigGenerated.h)Lines 95 to 114 in 49d6e0e
So when users install the CRAN binary, they get headers pre-configured for CRAN's environment, not their local setup. If we switched to using the compilation test for the manual macOS case, we'd need to change
ARMA_USE_OPENMPto be determined per-user at compile time rather than baked into the distributed headers. Otherwise users without OpenMP would get compilation errors.There's also the runtime version mismatch I mentioned earlier (Clang 14 vs 17), but that's a separate issue from the detection itself. In short, the manual tweak will run fine for basic OpenMP features, but certain constructs - like
schedule(dynamic)- will fail at runtime with the symbol error I mentioned earlier because the locally installedlibomp.dylib(Clang 17) doesn't match what CRAN's binaries were built against (Clang 14) and there's an underlying bug with Clang 17 and OpenMP.The current approach avoids both problems (hard coded headers and the toolchain mismatch) by leaving
ARMA_USE_OPENMPundefined for the Apple compiler toolchain. If we detect a different compiler, then the correct flags are set if the user compiles it locally.Oh I see. I really is more than a 2 x 2 matrix 😆
Maybe we have to recommend to macOS users compiling with RcppArmadillo to install it locally from source?
Would it help if wrapped this
in another user-level
#defineto accommodate or override the potential spill-over from CRAN for macOS users of the binary package?@eddelbuettel Wrapping it in a user-level override would give manual macOS users an escape hatch without breaking the CRAN binary for everyone else. Though, users can already do this by defining
-DARMA_USE_OPENMPin their~/.R/Makevars. So the manual override path exists already?
The interesting option is to make this automatic by detecting local OpenMP availability at compile time:
#if !defined(ARMA_USE_OPENMP) #if defined(__APPLE__) && defined(_OPENMP) // User has OpenMP available, but check if they want it disabled #ifndef RCPPARMA_MACOS_DISABLE_OPENMP #define ARMA_USE_OPENMP 1 #else #define ARMA_DONT_USE_OPENMP 1 #endif #else @ARMA_HAVE_OPENMP@ #endif #endif
This checks for macOS via
__APPLE__and compiler OpenMP support via_OPENMP. It would automatically enable OpenMP for manual macOS setups, with an opt-out via-DRCPPARMA_MACOS_DISABLE_OPENMPif users hit runtime version issues.
However, this opens up a complication with the inline plugin architecture if folks are using
Rcpp::sourceCpp(). Currently,inlineCxxPlugin()(lines 20-23) explicitly disables OpenMP flags on macOS:ismacos <- Sys.info()[["sysname"]] == "Darwin" openmpflag <- if (ismacos) "" else "$(SHLIB_OPENMP_CFLAGS)"
This prevents compile-time errors from RcppArmadilloConfig.h lines 113-121 when users don't have OpenMP installed.
If we enable automatic detection in the headers, should the plugin check also move to detecting whether OpenMP headers and runtime are actually present on macOS or that the define and other flags exists rather than blanket disabling it? We'd need to search for the presence of
/usr/local/include/omp.hand/usr/local/lib/libomp.dylibat runtime or parse thePKG_CPPFLAGSenvironment variable for-DARMA_USE_OPENMPas well as-Xclang -fopenmpandPKG_LIBSfor-lomp.Thoughts?
Excellent, really excellent. I like the snippet to be added to the config already altered by
configureso that is a 'yes, can do'.I think the plugin can be simplified by dropping the Apple case. We are doing all this because generally we can rely on
$(SHLIB_OPENMP_CFLAGS). Or do you think that too needs an override / a wrapping into#if ARMA_USE_OPENMPor alike?@coatless: Ok with a bit of delay (my bad ...) I finally got around and committed your two suggestions / the two items we arrived at here (along with a slight quietener for windows). This is in a new branch at GitHub and a new (draft) PR #497.
Any chance you can give this a spin in the next few days?
In the past we somewhat complicated install-time checks for OpenMP, initially in shell scripts later in
configure/autoconfcode. Yet these days we pass down to our client programs to just rely on what R supplies. The defaultMakevarshas (in both cases, .win or not) the lineswhich are being supplied by R. I think we can do the same for RcppArmadillo and skip the logic in
configure.ac. (Apart maybe from the quick test of whether we can/cannot compile against OpenMP) and also take advantage of what R has to offer.@coatless What do you think re macOS? Will R reflect correct what the user / has not installed, and can we rely on
SHLIB_OPENMP_CXXFLAGS?