[Dataset] macOS JNI library crashes the JVM on the first JSON read #1429

Describe the bug, including details regarding any error messages, version, and platform.

The macOS JNI library that the RC workflow builds (jni-macos-aarch_64) crashes the JVM the first time a dataset reads a JSON file. TestFileSystemDataset#testBaseJsonRead crashes every time on an Apple silicon Mac. The crash happens inside new FileSystemDatasetFactory(...), while it reads the two-row JSON file the test writes:

#  SIGSEGV (0xb) at pc=0x00000001644614e0, pid=57760, tid=63499
# Problematic frame:
# C  [jnilib-15442808047630463497.tmp+0x2c94e0]  arrow::Result<simdjson::arm64::ondemand::object> arrow::internal::ResolveSimdjsonResult<simdjson::arm64::ondemand::object>(simdjson::simdjson_result<simdjson::arm64::ondemand::object>, std::__1::basic_string_view<char, std::__1::char_traits<char>>)+0x2c

siginfo: si_signo: 11 (SIGSEGV), si_code: 2 (SEGV_ACCERR), si_addr: 0x0000000367f6a070

Native frames:
C  [jnilib-...tmp+0x2c94e0]  arrow::internal::ResolveSimdjsonResult<simdjson::arm64::ondemand::object>(...)+0x2c
C  [jnilib-...tmp+0x1233a7c]  arrow::json::(anonymous namespace)::ParseImpl::ParseValue(simdjson::arm64::ondemand::value&)+0x1400
C  [jnilib-...tmp+0x12317d0]  arrow::json::(anonymous namespace)::ParseImpl::Parse(std::__1::shared_ptr<arrow::Buffer> const&)+0x2cc
C  [jnilib-...tmp+0x123e978]  arrow::json::(anonymous namespace)::ParseBlock(...)+0x408
C  [jnilib-...tmp+0x123de54]  arrow::json::(anonymous namespace)::DecodingOperator::operator()(...) const+0x48
...
C  [jnilib-...tmp+0x12dad8c]  std::__1::__thread_proxy<... arrow::internal::ThreadPool::LaunchWorkersUnlocked(int)::$_0>>(void*)+0x194

The program counter offset is the same in every run. The fault address changes, but it always ends in f6a070 (another run: si_addr: 0x000000033af6a070).

Why CI doesn't catch it

• Arrow C++ 26.0.0's JSON parser still uses RapidJSON. Arrow C++ main switched to simdjson in GH-51037: [C++] Replace RapidJSON with simdjson in JSON parser arrow#51038.
• The RC workflow builds the macOS library from Arrow C++ main. Except in the nightly, it builds the Linux libraries from the latest release.
• The dataset tests run only in the Binaries job, on Linux x86_64.
• So nothing tests the macOS library, which is the only one built from main on every run. jni-macos checks out main for every trigger, release tags included, so a release built now would ship this crash in the macOS arrow-dataset library.

What I ruled out

• Arrow C++ main on Linux. The nightly's Binaries job passes TestFileSystemDataset on x86_64 (run 41256646402). That run's jni-linux-aarch_64 library passes it on arm64 too: 18/18, twice, in an eclipse-temurin:17-jdk container on Apple silicon. So the simdjson parser works on arm64 Linux.
• The memory allocator. The crash is identical with ARROW_DEFAULT_MEMORY_POOL=system and with ARROW_DEFAULT_MEMORY_POOL=mimalloc.

A lead I couldn't verify

The macOS runner already has Homebrew's simdjson installed: brew update lists simdjson among the outdated formulae. ci/scripts/jni_macos_build.sh builds the bundled simdjson 4.6.4 (-Dsimdjson_SOURCE=BUNDLED). Other Brewfile packages, such as Boost, Snappy, Thrift and xsimd, are installed under /opt/homebrew, so /opt/homebrew/include is probably on the include path. That raises a possibility: arrow/json/parser.cc could be compiled against Homebrew's simdjson.h but linked with the bundled simdjson, and a version mismatch like that could crash this way. The workflow already uninstalls Homebrew's RE2 and Protobuf because their files can get mixed into the build, and #1415 was a similar problem with lz4. Two ways to confirm or rule this out:

• check which simdjson.h the build's compile_commands.json uses for parser.cc;
• uninstall Homebrew's simdjson (brew uninstall --ignore-dependencies simdjson) before the build.

To reproduce

gh run download 41256646402 -R apache/arrow-java -n jni-macos-aarch_64
tar -xzf jni-macos-aarch_64.tar.gz
mvn -Parrow-jni -pl dataset -am test \
  -Darrow.c.jni.dist.dir=$PWD/jni -Darrow.cpp.build.dir=$PWD/jni \
  -Dtest='TestFileSystemDataset#testBaseJsonRead' -Dsurefire.failIfNoSpecifiedTests=false

The test JVM dies with The forked VM terminated without properly saying goodbye, and dataset/hs_err_pid<pid>.log has the stack above.

Tested on:

• macOS 26.6.2 on Apple silicon, Temurin 17.0.17.
• arrow-java main (67ef5ab). The crash also happens with the arrow-java changes from GH-88: Support importing C Data Interface arrays with a non-zero offset #1419.
• Two macOS libraries:
  • from run 41233658912, built from Arrow C++ apache/arrow@b4a98b731d83: 5 of 5 runs crashed;
  • from the nightly run 41256646402, built from apache/arrow@d9a0ab4b088b: crashed.