GH-39295: [C++][Python] ConsumingDLPack on Arrays and Tensor by AntoinePrv · Pull Request #51122 · apache/arrow

AntoinePrv · 2026-09-01T14:05:46Z

Rationale for this change

There are already utilities to import tensor data from NumPy.
With DLPack standard, it will work across many tensor providers.

What changes are included in this PR?

Bind FixedShapedTensorArray::FromTensor in Python
Add ImportArrayVersionedFromDLPack and ImportTensorVersionedFromDLPack in C++
Add in Python Array.from_dlpack and Tensor.from_dlpack
Similar to the export, multidimensional tensor import require an explicit step through Tensor.
No Array::FromTensor or even FixedSizeListArray::FromTensor. IMHO it does not feel as necessary in this direction but open to anyone's take one it.

Are these changes tested?

Yes.

Are there any user-facing changes?

Yes, new APIs.

GitHub Issue: [C++][Python] DLPack implementation for Arrow Arrays and Tensors (consuming) #39295

github-actions · 2026-09-01T14:08:08Z

⚠️ GitHub issue #39295 has been automatically assigned in GitHub to PR creator.

AntoinePrv · 2026-09-03T17:25:31Z

@AlenkaF @rok @pitrou this is looking good

Copilot

🟡 Changes recommended

There is a likely C++ compile-breaking scoping issue in cpp/src/arrow/tensor.cc around the stride helper functions, plus several doc/API consistency issues to resolve.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR adds DLPack consumer support to Arrow Arrays/Tensors in C++ and exposes new Python factory APIs (Array.from_dlpack, Tensor.from_dlpack), enabling zero-copy (or requested-copy) imports from DLPack-producing libraries (e.g., NumPy, PyTorch).

Changes:

Add C++ DLPack consumers: arrow::dlpack::ImportArrayVersioned and arrow::dlpack::ImportTensorVersioned, plus supporting utilities and tests.
Expose Python APIs for importing via versioned DLPack capsules, and bind FixedShapeTensorArray::FromTensor.
Add Python test coverage for tensor/array import, copy vs zero-copy behavior, and unsupported cases.

File summaries

File	Description
python/pyarrow/tests/test_dlpack.py	Adds Python tests for `Tensor.from_dlpack` / `Array.from_dlpack`, including copy semantics and unsupported multidim array import.
python/pyarrow/tensor.pxi	Implements `Tensor.from_dlpack` and wires it to C++ `ImportTensorVersioned`.
python/pyarrow/includes/libarrow.pxd	Declares DLPack version struct/constant and new C++ import APIs for Cython bindings; adds `FixedShapeTensorArray::FromTensor`.
python/pyarrow/array.pxi	Implements `Array.from_dlpack` and adds `FixedShapeTensorArray.from_tensor` binding.
cpp/src/arrow/tensor.h	Declares `ComputeTensorSize` helper for stride/shape-based size computation.
cpp/src/arrow/tensor.cc	Defines `ComputeTensorSize` and adjusts internal stride helper placement/calls.
cpp/src/arrow/c/dlpack.h	Exposes compiled DLPack version constant and declares new import APIs.
cpp/src/arrow/c/dlpack.cc	Implements DLPack version constant and consumer-side import logic for arrays/tensors.
cpp/src/arrow/c/dlpack_test.cc	Adds comprehensive C++ tests for importing (ownership, copy vs share, dtype coverage, errors).

Review details

Suppressed comments (1)

cpp/src/arrow/c/dlpack.h:132

The parameter doc for ImportTensorVersioned says raw is an "Arrow array", but the function takes a DLManagedTensorVersioned* (a DLPack tensor).

/// \param[in] raw Arrow array

Files reviewed: 9/9 changed files
Comments generated: 5
Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot

🟡 Changes recommended

There are correctness and robustness issues in the new DLPack consumer path (notably error type mismatch for unsupported array layouts, plus missing input validation that can crash on malformed tensors).

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Files reviewed: 9/9 changed files
Comments generated: 4
Review effort level: Lite

Copilot

🟡 Changes recommended

There is a confirmed C++ compilation issue in tensor.cc, and the Python copy= contract isn’t enforced by the current from_dlpack implementations.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

cpp/src/arrow/tensor.cc:563

Tensor::is_row_major() / is_column_major() call internal::IsTensorStridesRowMajor/ColumnMajor, but those helpers are defined inside an anonymous namespace nested under arrow::internal (in this translation unit), so arrow::internal::IsTensorStridesRowMajor doesn't exist. This will fail to compile. Consider computing the expected row/column-major strides directly in these methods (or moving the helpers out of the anonymous namespace).

bool Tensor::is_row_major() const {
  return internal::IsTensorStridesRowMajor(type_, shape_, strides_);
}

Files reviewed: 9/9 changed files
Comments generated: 3
Review effort level: Lite

Copilot

🟡 Changes recommended

There are compile- and correctness-blocking issues in the C++ tensor stride helper linkage and in the Cython from_dlpack implementations (nogil assignment and unchecked capsule consumption marking).

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

cpp/src/arrow/tensor.cc:227

Tensor::is_row_major() / is_column_major() call internal::IsTensorStridesRowMajor/ColumnMajor, but those functions are currently only defined in an anonymous namespace (not arrow::internal), so this won’t compile. Define IsTensorStridesRowMajor and IsTensorStridesColumnMajor in namespace internal (next to IsTensorStridesContiguous) or revert the qualified calls.

}  // namespace
}  // namespace internal

namespace internal {

Files reviewed: 9/9 changed files
Comments generated: 3
Review effort level: Lite

pitrou · 2026-09-03T17:32:12Z

+                                      {data, nbytes}, default_cpu_memory_manager()));
+  } else {
+    const bool readonly = dl.is_readonly();
+    // Trick to keep DLPack data alive taken from `Buffer::FromVector`.


A new Buffer subclass would be cleaner than this IMHO.

That said, I understand that the subclassing strategy might not be scalable once we want to support non-CPU devices.

I does seem a bit overkill to subclass for that purpose.
What about adding Buffer::TakeOwnership(T&& container, uint8_t const* data, int64_t nbytes); (where data points into container)? I can see it being universally useful to a C++ user.

Why not, yes!

pitrou · 2026-09-07T08:07:55Z

+///            producer.
+/// \return An Arrow Array
+ARROW_EXPORT
+Result<std::shared_ptr<Array>> ImportArrayVersioned(DLManagedTensorVersioned* raw,


Is the "Versioned" suffix useful? We're not exposing any non-versioned API, and even then, we could just rely on function overloading.

Mostly to match the producer side ExportArrayVersioned, I feel it easier to match the importing and exporting functions that way.

pitrou · 2026-09-07T08:25:47Z

+  } else if (copy) {
+    ARROW_ASSIGN_OR_RAISE(buffer, MutableBuffer::CopyNonOwned(
+                                      {data, nbytes}, default_cpu_memory_manager()));
+  } else {


Presumably, a caller setting copy wants to get an exclusive mutable copy of the tensor.

Should we then inspect the DLPack flags for DLPACK_FLAG_BITMASK_IS_COPIED? In that case, a new copy doesn't need to be made, I think?

(also check that DLPACK_FLAG_BITMASK_READ_ONLY is not set?)

@pitrou I've been thinking with this and:

First we do not need the copy parameter in ImportArray because we already can pass copy to the producer (which is what we do in Python).

Then without copy I believe we have no use for DLPACK_FLAG_BITMASK_IS_COPIED.

What do you think, should we keep our copy parameter (currently unused) or always rely on the producer to copy?

If it's currently unused then let's remove it for now?

On the Python side, as you say, __dlpack__ is called first.
If there are some direct C/C++ uses of DLPack, then we may want to revisit later.

pitrou · 2026-09-07T08:41:57Z

+        x : object
+            The input object containing array data, following the DLPack
+            protocol (has a ``__dlpack__`` method).
+        device : tuple[enum.Enum, int], optional


PyArrow also has its own Device class, can we add a TODO and/or open an issue to support it here?

>>> mm = pa.default_cpu_memory_manager() >>> mm.device <pyarrow.Device: CPUDevice()> >>> mm.device.device_type <DeviceAllocationType.CPU: 1> >>> import pyarrow.cuda as cu >>> ctx = cu.Context() >>> ctx.device <pyarrow.Device: CudaDevice(device_number=0, name="NVIDIA GeForce GT 1030")> >>> ctx.device.device_type <DeviceAllocationType.CUDA: 2>

That would be outside the DLPack spec. Do you think its reasonable?

Well NumPy supports a string, PyTorch supports a torch.device, JAX supports a _jax.Device. I think PyArrow can support a PyArrow device :)

Copilot

🟡 Changes recommended

The new C++ DLPack consumer treats ndim == 0 tensors as having 0 elements, which breaks scalar tensor imports (should be 1 element).

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Files reviewed: 14/14 changed files
Comments generated: 1
Review effort level: Lite

Copilot

🟡 Changes recommended

There is at least one confirmed build-breaking issue in cpp/src/arrow/tensor.cc plus a couple of missing error-handling checks that could lead to undefined behavior or double-deletion in DLPack import paths.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (3)

cpp/src/arrow/tensor.cc:566

Tensor::is_row_major() / Tensor::is_column_major() call internal::IsTensorStridesRowMajor/ColumnMajor, but those helpers are not defined in arrow::internal (they are currently in an unnamed namespace), so this won’t compile. Implement these checks inline using internal::ComputeRowMajorStrides / internal::ComputeColumnMajorStrides (or alternatively move the helpers out of the unnamed namespace into arrow::internal).

  return internal::IsTensorStridesRowMajor(type_, shape_, strides_);
}

bool Tensor::is_column_major() const {
  return internal::IsTensorStridesColumnMajor(type_, shape_, strides_);

python/pyarrow/array.pxi:2319

PyCapsule_SetName() can fail (returns non-zero and sets an exception). If that happens, this code will still proceed to call the C++ importer with a pending Python error and with the capsule possibly not marked as consumed, which risks double-deletion when the capsule is later GC’d. Check the return value and abort on failure.

        # Mark the capsule as consumed so its destructor does not also invoke the deleter.
        # ImportArrayVersionedFromDLPack will take ownership even if it errors (calling
        # the deleter in that case).
        PyCapsule_SetName(pycapsule, "used_dltensor_versioned")
        with nogil:

python/pyarrow/tensor.pxi:348

PyCapsule_SetName() can fail (returns non-zero and sets an exception). If that happens, this code will still proceed to call the C++ importer with a pending Python error and with the capsule possibly not marked as consumed, which risks double-deletion when the capsule is later GC’d. Check the return value and abort on failure.

        # Mark the capsule as consumed so its destructor does not also invoke the deleter.
        # ImportTensorVersionedFromDLPack will take ownership even if it errors (calling
        # the deleter in that case).
        PyCapsule_SetName(pycapsule, "used_dltensor_versioned")
        with nogil:

Files reviewed: 14/14 changed files
Comments generated: 1
Review effort level: Lite

AlenkaF · 2026-09-10T08:14:17Z

@AntoinePrv I see the last comment from @pitrou has been addressed and feel this is ready to be merged if I am not missing anything.

Copilot

Copilot was unable to review this pull request because the user who requested the review is ineligible. To be eligible to request a review, you need a paid Copilot license, or your organization must enable Copilot code review.

pitrou

Thanks for the updates @AntoinePrv ! This LGTM, we can merge if CI is green.

pitrou · 2026-09-10T15:04:39Z

CI failures are unrelated.

github-actions Bot added the awaiting review Awaiting review label Sep 1, 2026

github-actions Bot added Component: C++ Component: Python labels Sep 1, 2026

AntoinePrv force-pushed the dl-consume-2 branch 2 times, most recently from 8d1eca2 to 2fe859d Compare September 3, 2026 16:16

AntoinePrv marked this pull request as ready for review September 3, 2026 17:25

Copilot AI lite review requested due to automatic review settings September 3, 2026 17:25

AntoinePrv requested review from AlenkaF, pitrou, raulcd and rok as code owners September 3, 2026 17:25

Copilot started reviewing on behalf of AntoinePrv September 3, 2026 17:25 View session

Copilot AI reviewed Sep 3, 2026

View reviewed changes

Comment thread cpp/src/arrow/tensor.cc

Comment thread cpp/src/arrow/c/dlpack.h Outdated

Comment thread python/pyarrow/array.pxi

Comment thread python/pyarrow/tensor.pxi

Comment thread python/pyarrow/tensor.pxi Outdated

Copilot AI review requested due to automatic review settings September 4, 2026 07:53

Copilot started reviewing on behalf of AntoinePrv September 4, 2026 07:54 View session

Copilot AI reviewed Sep 4, 2026

View reviewed changes

Comment thread cpp/src/arrow/c/dlpack.cc

Comment thread cpp/src/arrow/c/dlpack.cc Outdated

Comment thread cpp/src/arrow/c/dlpack.cc Outdated

Comment thread python/pyarrow/array.pxi

Copilot AI review requested due to automatic review settings September 4, 2026 08:55

AntoinePrv force-pushed the dl-consume-2 branch from c940592 to 625aab0 Compare September 4, 2026 08:55

Copilot started reviewing on behalf of AntoinePrv September 4, 2026 08:57 View session

Copilot AI reviewed Sep 4, 2026

View reviewed changes

Comment thread cpp/src/arrow/c/dlpack.cc Outdated

Comment thread python/pyarrow/array.pxi

Comment thread python/pyarrow/tensor.pxi

Copilot AI review requested due to automatic review settings September 4, 2026 09:12

AntoinePrv force-pushed the dl-consume-2 branch from 625aab0 to 496604c Compare September 4, 2026 09:12

Copilot started reviewing on behalf of AntoinePrv September 4, 2026 09:13 View session

Copilot AI reviewed Sep 4, 2026

View reviewed changes

Comment thread python/pyarrow/array.pxi

Comment thread python/pyarrow/tensor.pxi

Comment thread python/pyarrow/tests/test_dlpack.py

AntoinePrv mentioned this pull request Sep 4, 2026

GH-38868: [Python] Add dlpack producer to FixedShapeTensorArray/Scalar #51159

Merged

pitrou reviewed Sep 7, 2026

View reviewed changes

github-actions Bot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Sep 7, 2026

AntoinePrv added 12 commits September 9, 2026 10:24

Add Buffer::TakeOwnership

3ab9add

Improve tests

4d20306

Remove consumer copy param

e343b9e

Document mutable buffers in Buffer::FromString / Vec

6d0aa95

Fix doc

9a56ffe

Add FixedShapeTensorArray.from_tensor test

55ade09

Fix mutability test

164676b

Typos

b2095a1

Update DLPack documentation

c52f747

Add FixedShapeTensorArray.from_dlpack

98d4198

Add missing tests

d8ea00c

Handle row major strides

381e5ea

Copilot AI review requested due to automatic review settings September 9, 2026 08:25

AntoinePrv force-pushed the dl-consume-2 branch from b9eb12e to 381e5ea Compare September 9, 2026 08:25

Copilot started reviewing on behalf of AntoinePrv September 9, 2026 08:27 View session

Copilot AI reviewed Sep 9, 2026

View reviewed changes

Comment thread cpp/src/arrow/c/dlpack.cc Outdated

Fix 0-D tensor

ac31c1e

Copilot AI review requested due to automatic review settings September 9, 2026 09:26

Copilot started reviewing on behalf of AntoinePrv September 9, 2026 09:31 View session

Copilot AI reviewed Sep 9, 2026

View reviewed changes

Comment thread cpp/src/arrow/c/dlpack.cc

pitrou reviewed Sep 10, 2026

View reviewed changes

Comment thread cpp/src/arrow/c/dlpack.cc Outdated

Fix typo

728079d

Copilot AI review requested due to automatic review settings September 10, 2026 13:07

Copilot AI reviewed Sep 10, 2026

pitrou approved these changes Sep 10, 2026

View reviewed changes

pitrou merged commit 39706ed into apache:main Sep 10, 2026
58 of 63 checks passed

pitrou removed the awaiting committer review Awaiting committer review label Sep 10, 2026

pitrou mentioned this pull request Sep 10, 2026

[C++][Python] DLPack implementation for Arrow Arrays and Tensors (consuming) #39295

Closed

Uh oh!

Conversation

AntoinePrv commented Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions Bot commented Sep 1, 2026

Uh oh!

AntoinePrv commented Sep 3, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Suppressed comments (1)

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Suppressed comments (1)

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Suppressed comments (1)

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

AntoinePrv commented Sep 1, 2026 •

edited

Loading