GH-51245: [C++][Gandiva] Add support for LLVM 23.1.0

[raulcd]

Rationale for this change

Some of our conda-cpp jobs are failing due to the newer version on LLVM on conda.

What changes are included in this PR?

Add 23.1 to the list of supported versions and adapt to new API on jit_builder.setObjectLinkingLayerCreator.
The related API change:

• [ORC] Add MemMgr arg to LLJITBuilder::ObjectLinkingLayerCreator. llvm/llvm-project#192214

Are these changes tested?

Yes, tested locally and will force a docker rebuild to test via crossbow.

Are there any user-facing changes?

No

• GitHub Issue: [C++][Gandiva] Add support for LLVM 23.1 #51245

[raulcd]

@gibthub-actions crossbow submit test-conda-cpp

[Copilot]

🟢 Approval recommended

The functional LLVM 23.1 compatibility updates are small and scoped, with only a minor maintainability suggestion remaining for the Docker cache-bust step.

Pull request overview

Updates Gandiva’s LLVM integration to accept LLVM 23.1.x (needed for newer conda-provided LLVM) and adapts the ORC JITLink layer-creator API changes introduced in LLVM 23.

Changes:

• Add LLVM 23.1 to the CMake-supported LLVM versions list.
• Update LLJITBuilder::setObjectLinkingLayerCreator usage for LLVM >= 23.
• Bust the conda-cpp Docker image cache to trigger a rebuild in CI.

File summaries

File	Description
cpp/src/gandiva/engine.cc	Adjusts ORC JITLink layer-creator callback signature for LLVM 23 compatibility.
cpp/CMakeLists.txt	Adds LLVM 23.1 to the supported LLVM version list used by LLVM discovery.
ci/docker/conda-cpp.dockerfile	Adds a cache-busting layer to force a CI image rebuild.

Review details

• Files reviewed: 3/3 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[raulcd]

@github-actions crossbow submit test-conda-cpp

[github-actions]

Revision: 5e73a38

Submitted crossbow builds: ursacomputing/crossbow @ actions-c3cf226205

Task	Status
test-conda-cpp

[raulcd]

@github-actions crossbow submit test-conda-cpp

[github-actions]

Revision: c25e5c4

Submitted crossbow builds: ursacomputing/crossbow @ actions-c0ff0e55e0

Task	Status
test-conda-cpp

[Copilot]

🟡 Changes recommended

It includes CI/docker and conda-environment changes that should be tightened (remove the no-op Docker layer and keep an upper bound on llvmdev to avoid future breakage).

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

ci/docker/conda-cpp.dockerfile:31

• The added RUN echo "Force rebuild" creates an extra image layer and keeps a cache-busting hack in the Dockerfile. Prefer triggering rebuilds via CI/build args rather than committing a no-op RUN step.

RUN echo "Force rebuild"

• Files reviewed: 4/4 changed files
• Comments generated: 1
• Review effort level: Lite

[raulcd]

@github-actions crossbow submit test-conda-cpp

[github-actions]

Revision: 10e83b4

Submitted crossbow builds: ursacomputing/crossbow @ actions-f1e02ee081

Task	Status
test-conda-cpp

[Copilot]

🟡 Changes recommended

CI/environment robustness and test-failure reporting should be tightened (restore an llvmdev upper bound, drop the redundant Docker cache-bust line, and make the new shared test helper fail-fast on Status errors).

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (2)

cpp/src/gandiva/tests/projector_test.cc:2133

• Same as above: if Evaluate() fails, continuing can hide the root cause and may access incomplete outputs. Use ASSERT_OK(status) so failures are reported at the correct point.

  status = projector->Evaluate(*in_batch, pool, &outputs);
  EXPECT_TRUE(status.ok());

ci/conda_env_gandiva.txt:19

• Removing the llvmdev upper bound makes this environment vulnerable to the next LLVM major upgrade in conda, which will break builds again until Arrow/Gandiva adds support. Since CMake currently only advertises support up to 23.1, keep an upper bound (<24) to prevent unexpected CI breakage.

llvmdev

• Files reviewed: 5/5 changed files
• Comments generated: 2
• Review effort level: Lite

[raulcd]

@github-actions crossbow submit test-conda-cpp

[github-actions]

Revision: e32cc10

Submitted crossbow builds: ursacomputing/crossbow @ actions-5c21eae4e8

Task	Status
test-conda-cpp

[raulcd]

We are currently pinning LLVM<23 at main. I've rebased and my immediate CI failure issue has been solved. I am not an expert on Gandiva or LLVMA and as I am getting some test errors I just close this as an investigation. I might come back to it later in the future but I am unassigning the issue if someone wants to investigate.

[raulcd]

I am not an expert here but should we use a llvm::jitlink::JITLinkMemoryManager& memory_manager here?

-#  if LLVM_VERSION_MAJOR >= 21
+#  if LLVM_VERSION_MAJOR >= 23
+    jit_builder.setObjectLinkingLayerCreator(
+        [](llvm::orc::ExecutionSession& ES,
+           llvm::jitlink::JITLinkMemoryManager& memory_manager) {
+          return std::make_unique<llvm::orc::ObjectLinkingLayer>(ES, memory_manager);
+        });
+#  elif LLVM_VERSION_MAJOR >= 21

@dmitry-chirkov-dremio @lriggs @akravchukdremio @xxlaykxx @kou

[raulcd]

sorry, I didn't mean to post this comment (wrong click) as I was trying to find what the problems were with the test failures but if someone with more expertise here wants to follow it up :)