[Python][ppc64le][cuda] pytest segfault - test_cuda.py/test_foreign_buffer

[prashantkhoje]

Hello everyone,
I tried to build and test pyarrow on ppc64le platform with CUDA.
I encounter segmentation fault at test_cuda.py/test_foreign_buffer#L292.
If i comment out the test_foreign_buffer, then rest of the tests pass successfully.

Has anyone experienced the same either on ppc or x86-64?
I'm sharing steps i followed and log.

# STEPS

---

mkdir p11-arrow-repo
cd p11-arrow-repo
git clone https://github.com/apache/arrow.git
cd arrow
git checkout apache-arrow-7.0.0
# Not available: benchmark (conda_env_cpp.txt) - Removed for now.
vim ci/conda_env_cpp.txt
git diff ci/conda_env_cpp.txt
cd ..

conda create -y -n p11-arrow -c conda-forge
conda activate p11-arrow

conda install -y -c conda-forge \
    --file arrow/ci/conda_e
[p11-arrow.txt](https://github.com/apache/arrow/files/8215513/p11-arrow.txt)
nv_unix.txt \
    --file arrow/ci/conda_env_cpp.txt \
    --file arrow/ci/conda_env_python.txt \
    --file arrow/ci/conda_env_gandiva.txt \
    compilers \
    python=3.9 \
    pandas

conda install -y -c conda-forge cudatoolkit=11.2

pushd arrow
git submodule init
git submodule update
export PARQUET_TEST_DATA="${PWD}/cpp/submodules/parquet-testing/data"
export ARROW_TEST_DATA="${PWD}/testing/data"
popd

# CPP
export ARROW_HOME=$CONDA_PREFIX
export LD_LIBRARY_PATH=$CONDA_PREFIX/lib

mkdir arrow/cpp/build
pushd arrow/cpp/build

cmake -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \
      -DCMAKE_INSTALL_LIBDIR=lib \
      -DCMAKE_BUILD_TYPE=release \
      -DARROW_WITH_BZ2=ON \
      -DARROW_WITH_ZLIB=ON \
      -DARROW_WITH_ZSTD=ON \
      -DARROW_WITH_LZ4=ON \
      -DARROW_WITH_SNAPPY=ON \
      -DARROW_WITH_BROTLI=ON \
      -DARROW_PARQUET=ON \
      -DARROW_PYTHON=ON \
      -DARROW_BUILD_TESTS=ON \
      -DARROW_ORC=ON \
      -DARROW_DATASET=ON \
      -DARROW_HDFS=ON \
      -DARROW_CUDA=ON -DARROW_EXTRA_ERROR_CONTEXT=ON ..

make -j 4
make install
ctest
popd

# Python
pushd arrow/python
export PYARROW_WITH_PARQUET=1
export PYARROW_WITH_CUDA=1
export PYARROW_WITH_ORC=1
export PYARROW_WITH_DATASET=1
export PYARROW_WITH_HDFS=1
python setup.py build_ext --inplace
python setup.py install
pip install -e . --no-build-isolation
pytest -v
popd

Thank you,
Prashant

[prashantkhoje]

p11-arrow.txt

[prashantkhoje]

@leofang, @jakirkham, Have you encountered the failure mentioned here?
I don't have x86 with gpu to compare.

[pitrou]

@prashantkhoje Sorry for the delay. Can you try to get a gdb backtrace from the coredump file?

[pitrou]

We're now seeing the same crashes on the mainline CUDA Python CI as well. The test is inherently crashy as it's trying to access an invalid pointer.

[pitrou]

Issue resolved by pull request 51206
#51206