[C++][Python] `_export_to_c` segmentation fault for `binary_view` array

[drain99]

Describe the bug, including details regarding any error messages, version, and platform.

_export_to_c seems to cause segmentation fault for binary_view array constructed as follows:
ChunkedArray(binary...) --combine--> BinaryArray --cast--> BinaryViewArray

Python version: 3.11.13
PyArrow version: 23.0.1

Reproducer below

import ctypes
import pyarrow as pa

# C Data Interface layouts
class ArrowSchema(ctypes.Structure):
    _fields_ = [
        ("format", ctypes.c_char_p),
        ("name", ctypes.c_char_p),
        ("metadata", ctypes.c_char_p),
        ("flags", ctypes.c_int64),
        ("n_children", ctypes.c_int64),
        ("children", ctypes.POINTER(ctypes.c_void_p)),
        ("dictionary", ctypes.c_void_p),
        ("release", ctypes.c_void_p),
        ("private_data", ctypes.c_void_p),
    ]

class ArrowArray(ctypes.Structure):
    _fields_ = [
        ("length", ctypes.c_int64),
        ("null_count", ctypes.c_int64),
        ("offset", ctypes.c_int64),
        ("n_buffers", ctypes.c_int64),
        ("n_children", ctypes.c_int64),
        ("buffers", ctypes.POINTER(ctypes.c_void_p)),
        ("children", ctypes.POINTER(ctypes.c_void_p)),
        ("dictionary", ctypes.c_void_p),
        ("release", ctypes.c_void_p),
        ("private_data", ctypes.c_void_p),
    ]

c_array = ArrowArray()
c_schema = ArrowSchema()

arrs = [
    # exporting this is fine
    pa.array([b'a', b'e', b'a', b'e'], type=pa.binary_view()),
    # this also works fine
    pa.chunked_array([
        pa.array([b'a', b'e'], type=pa.binary_view()),
        pa.array([b'a', b'e'], type=pa.binary_view())
    ]).combine_chunks(),
    # chunked(binary...) -> binary -> binary_view segfaults
    pa.chunked_array([
        pa.array([b'a', b'e'], type=pa.binary()),
        pa.array([b'a', b'e'], type=pa.binary())
    ]).combine_chunks().cast(pa.binary_view()),
]
print(arrs)
print([arr.validate(full=True) for arr in arrs])
arrs[0]._export_to_c(ctypes.addressof(c_array),ctypes.addressof(c_schema),)
arrs[1]._export_to_c(ctypes.addressof(c_array),ctypes.addressof(c_schema),)
# this fails
arrs[2]._export_to_c(ctypes.addressof(c_array),ctypes.addressof(c_schema),)

Segmentation fault (core dumped)

Component(s)

Python

[AlenkaF]

Thank you for opening the issue!
I can reproduce the segfault locally, the repr provided is very clear. I did a bit of looking into the issue and asked Copilot for some help. What I found was that the segfault only happens for short strings and works for long strings, for example:

In [2]: long = pa.chunked_array([
   ...:     pa.array([b'a' * 13, b'e' * 13], type=pa.binary()),
   ...:     pa.array([b'a' * 13, b'e' * 13], type=pa.binary()),
   ...: ]).combine_chunks().cast(pa.binary_view())

In [3]: long._export_to_c(ctypes.addressof(c_array), ctypes.addressof(c_schema))
   ...: print("Long strings: OK")
   ...: 
Long strings: OK

In [4]: short = pa.chunked_array([
   ...:     pa.array([b'a', b'e'], type=pa.binary()),
   ...:     pa.array([b'a', b'e'], type=pa.binary()),
   ...: ]).combine_chunks().cast(pa.binary_view())

In [5]: short._export_to_c(ctypes.addressof(c_array), ctypes.addressof(c_schema))
   ...: print("Short strings: OK")  # never reached
[1]    18460 segmentation fault  ipython

Copilot is pointing out that the bug is in the scalar_cast_string.cc implementation where the extra data buffer is dropped because all_entries_are_inline is True

arrow/cpp/src/arrow/compute/kernels/scalar_cast_string.cc

Lines 465 to 467 in e8b7b4e

	if (all_entries_are_inline) {
	output->buffers[2] = nullptr;
	}

So this would be a bug in the cast kernel. We might also update the bridge file

arrow/cpp/src/arrow/c/bridge.cc

Lines 606 to 614 in e8b7b4e

	if (need_variadic_buffer_sizes) {
	auto variadic_buffers = std::span(data->buffers).subspan(2);
	export_.variadic_buffer_sizes_.resize(variadic_buffers.size());
	size_t i = 0;
	for (const auto& buf : variadic_buffers) {
	export_.variadic_buffer_sizes_[i++] = buf->size();
	}
	export_.buffers_.back() = export_.variadic_buffer_sizes_.data();
	}

guarding against null-pointer variadic buffers.

cc @pitrou

[pitrou]

@AlenkaF Thanks for the analysis. I agree it's probably not ok to have null variadic buffers, we should instead resize the buffers in the cast kernel's output.

We should also update our array validation routines to ensure that null variadic buffers don't pass validation (i.e., in the snippet above, short.validate() should fail but it currently succeeds).

[pitrou]

@andishgar @fenfeng9 @Periecle Is one of you interested in working on this? While the reproducer is in Python, the bug is most probably in the C++ codebase.

[fenfeng9]

@pitrou Sure, I'd be happy to take this one.

[fenfeng9]

A smaller reproduction case:
Casting short binary values to binary_view produces an array whose third buffer slot is None ([None, 32, None]).

And validate(full=True) still succeeds, but _export_to_c segfaults when exporting that null variadic buffer slot.

Reproduce

import ctypes
import platform
import pyarrow as pa


if __name__ == "__main__":
    print(f"Python {platform.python_version()}, PyArrow {pa.__version__}", flush=True)
    print(flush=True)

    arr = pa.array([b"a", b"e"], type=pa.binary())
    print("binary array:", flush=True)
    print(arr, flush=True)
    print(
        f"binary buffers      : {[None if buf is None else buf.size for buf in arr.buffers()]}",
        flush=True,
    )

    arr = arr.cast(pa.binary_view())
    print()
    print("after cast to binary_view:", flush=True)
    print(arr, flush=True)
    print(
        f"binary_view buffers : {[None if buf is None else buf.size for buf in arr.buffers()]}",
        flush=True,
    )

    arr.validate(full=True)
    print("validate(full=True) : OK", flush=True)

    print("_export_to_c        : calling", flush=True)
    out = ctypes.create_string_buffer(1024)
    arr._export_to_c(ctypes.addressof(out))
    print("_export_to_c        : OK", flush=True)

Result

Python 3.12.13, PyArrow 24.0.0

binary array:
[
  61,
  65
]
binary buffers      : [None, 12, 2]

after cast to binary_view:
[
  61,
  65
]
binary_view buffers : [None, 32, None]
validate(full=True) : OK
_export_to_c        : calling

[andishgar]

@pitrou

This is the minimum C++ code to produce the error:

TEST(CdataInterface, BinaryView) {
  arrow::BinaryViewBuilder builder;
  ASSERT_OK(builder.Append("1"));
  ASSERT_OK_AND_ASSIGN(auto array_0, builder.Finish());
  array_0->data()->buffers.push_back(nullptr);
  struct ArrowArray array_out;
  struct ArrowSchema schema_out;
  ASSERT_OK(arrow::ExportArray(*array_0, &array_out, &schema_out));
}

The root of the problem is the line below, which assumes all data buffers of BinaryView are non‑null:

arrow/cpp/src/arrow/c/bridge.cc

Line 611 in e8b7b4e

export_.variadic_buffer_sizes_[i++] = buf->size();

I have a question: as far as I have read the specification, it does not mention whether the data buffers of BinaryViewType can be null or not. Is there any clarification for that in the Arrow specification?

[andishgar]

@fenfeng9, I'm aware that you submitted a PR. I don't intend to interfere it - I'm only asking to clarify the memory layout of BinaryView.

[fenfeng9]

@andishgar Thanks for looking into this, and no worries at all.

My current understanding is that, for all-inline BinaryView / Utf8View values, no variadic buffer is actually needed.

In practice, I found that Arrow can currently produce/accept both of these layouts:

[validity, views]
[validity, views, zero-size variadic data buffer]

For example, the normal C++ BinaryViewBuilder::Append() path produces the first form, while this PyArrow construction path produces the second form:

import pyarrow as pa

arr = pa.array([b"ab", b"cd", b"ef"], type=pa.binary_view())
print([None if b is None else b.size for b in arr.buffers()])

[None, 48, 0]

The extra zero-size buffer seems to come from the PyArrow conversion path reserving data space before appending the value:

arrow/python/pyarrow/src/arrow/python/python_to_arrow.cc

Lines 770 to 775 in 16fe342

	// Since we don't know the varying length input size in advance, we need to
	// reserve space in the value builder one by one. ReserveData raises CapacityError
	// if the value would not fit into the array.
	ARROW_RETURN_NOT_OK(this->primitive_builder_->ReserveData(view_.size));
	this->primitive_builder_->UnsafeAppend(view_.bytes,
	static_cast<OffsetType>(view_.size));

For short values, the data is still encoded inline, so the reserved heap block is unused and later becomes a zero-size variadic data buffer.

For this case:

[validity, views, null variadic buffer slot]

I also don't see the spec explicitly clarifying this, but I tend to think this should be considered an invalid state.