GH-51090: [CI][C++] Fix shellcheck errors in cpp/build-support/fuzzing/generate_corpuses.sh

[hiroyuki-sato]

Rationale for this change

This is the sub issue #44748.

• SC2046: Quote this to prevent word splitting.
• SC2128: Expanding an array without an index only gives the element in the index 0.
• SC2086: Double quote to prevent globbing and word splitting.

In generate_corpuses.sh line 32:
ARROW_ROOT=$(cd $(dirname "$BASH_SOURCE")/../../..; pwd)
                ^-----------------------^ SC2046 (warning): Quote this to prevent word splitting.
                           ^----------^ SC2128 (warning): Expanding an array without an index only gives the first element.


In generate_corpuses.sh line 43:
${OUT}/arrow-ipc-generate-fuzz-corpus -stream ${CORPUS_DIR}
^----^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean:
"${OUT}"/arrow-ipc-generate-fuzz-corpus -stream ${CORPUS_DIR}


In generate_corpuses.sh line 45:
IPC_INTEGRATION_FILES=$(find ${ARROW_ROOT}/testing/data/arrow-ipc-stream/integration -name "*.stream")
                             ^-----------^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean:
IPC_INTEGRATION_FILES=$(find "${ARROW_ROOT}"/testing/data/arrow-ipc-stream/integration -name "*.stream")


In generate_corpuses.sh line 49:
cp --backup=numbered ${IPC_INTEGRATION_FILES} ${CORPUS_DIR}
                     ^----------------------^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean:
cp --backup=numbered "${IPC_INTEGRATION_FILES}" ${CORPUS_DIR}


In generate_corpuses.sh line 50:
${ARROW_CPP}/build-support/fuzzing/pack_corpus.py ${CORPUS_DIR} ${OUT}/arrow-ipc-stream-fuzz_seed_corpus.zip
^----------^ SC2086 (info): Double quote to prevent globbing and word splitting.
                                                                ^----^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean:
"${ARROW_CPP}"/build-support/fuzzing/pack_corpus.py ${CORPUS_DIR} "${OUT}"/arrow-ipc-stream-fuzz_seed_corpus.zip


In generate_corpuses.sh line 53:
${OUT}/arrow-ipc-generate-fuzz-corpus -file ${CORPUS_DIR}
^----^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean:
"${OUT}"/arrow-ipc-generate-fuzz-corpus -file ${CORPUS_DIR}


In generate_corpuses.sh line 54:
IPC_INTEGRATION_FILES=$(find ${ARROW_ROOT}/testing/data/arrow-ipc-stream/integration -name "*.arrow_file")
                             ^-----------^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean:
IPC_INTEGRATION_FILES=$(find "${ARROW_ROOT}"/testing/data/arrow-ipc-stream/integration -name "*.arrow_file")


In generate_corpuses.sh line 56:
cp --backup=numbered ${IPC_INTEGRATION_FILES} ${CORPUS_DIR}
                     ^----------------------^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean:
cp --backup=numbered "${IPC_INTEGRATION_FILES}" ${CORPUS_DIR}


In generate_corpuses.sh line 57:
${ARROW_CPP}/build-support/fuzzing/pack_corpus.py ${CORPUS_DIR} ${OUT}/arrow-ipc-file-fuzz_seed_corpus.zip
^----------^ SC2086 (info): Double quote to prevent globbing and word splitting.
                                                                ^----^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean:
"${ARROW_CPP}"/build-support/fuzzing/pack_corpus.py ${CORPUS_DIR} "${OUT}"/arrow-ipc-file-fuzz_seed_corpus.zip


In generate_corpuses.sh line 60:
${OUT}/arrow-ipc-generate-tensor-fuzz-corpus -stream ${CORPUS_DIR}
^----^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean:
"${OUT}"/arrow-ipc-generate-tensor-fuzz-corpus -stream ${CORPUS_DIR}


In generate_corpuses.sh line 61:
${ARROW_CPP}/build-support/fuzzing/pack_corpus.py ${CORPUS_DIR} ${OUT}/arrow-ipc-tensor-stream-fuzz_seed_corpus.zip
^----------^ SC2086 (info): Double quote to prevent globbing and word splitting.
                                                                ^----^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean:
"${ARROW_CPP}"/build-support/fuzzing/pack_corpus.py ${CORPUS_DIR} "${OUT}"/arrow-ipc-tensor-stream-fuzz_seed_corpus.zip


In generate_corpuses.sh line 66:
${OUT}/parquet-arrow-generate-fuzz-corpus ${CORPUS_DIR}
^----^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean:
"${OUT}"/parquet-arrow-generate-fuzz-corpus ${CORPUS_DIR}


In generate_corpuses.sh line 68:
cp ${ARROW_CPP}/submodules/parquet-testing/data/*.parquet ${CORPUS_DIR}
   ^----------^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean:
cp "${ARROW_CPP}"/submodules/parquet-testing/data/*.parquet ${CORPUS_DIR}


In generate_corpuses.sh line 69:
cp ${ARROW_CPP}/submodules/parquet-testing/bad_data/*.parquet ${CORPUS_DIR}
   ^----------^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean:
cp "${ARROW_CPP}"/submodules/parquet-testing/bad_data/*.parquet ${CORPUS_DIR}


In generate_corpuses.sh line 70:
${ARROW_CPP}/build-support/fuzzing/pack_corpus.py ${CORPUS_DIR} ${OUT}/parquet-arrow-fuzz_seed_corpus.zip
^----------^ SC2086 (info): Double quote to prevent globbing and word splitting.
                                                                ^----^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean:
"${ARROW_CPP}"/build-support/fuzzing/pack_corpus.py ${CORPUS_DIR} "${OUT}"/parquet-arrow-fuzz_seed_corpus.zip


In generate_corpuses.sh line 75:
${OUT}/parquet-generate-encoding-fuzz-corpus ${CORPUS_DIR}
^----^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean:
"${OUT}"/parquet-generate-encoding-fuzz-corpus ${CORPUS_DIR}


In generate_corpuses.sh line 76:
${ARROW_CPP}/build-support/fuzzing/pack_corpus.py ${CORPUS_DIR} ${OUT}/parquet-encoding-fuzz_seed_corpus.zip
^----------^ SC2086 (info): Double quote to prevent globbing and word splitting.
                                                                ^----^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean:
"${ARROW_CPP}"/build-support/fuzzing/pack_corpus.py ${CORPUS_DIR} "${OUT}"/parquet-encoding-fuzz_seed_corpus.zip


In generate_corpuses.sh line 84:
${OUT}/arrow-csv-generate-fuzz-corpus ${CORPUS_DIR}
^----^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean:
"${OUT}"/arrow-csv-generate-fuzz-corpus ${CORPUS_DIR}


In generate_corpuses.sh line 86:
cp ${ARROW_ROOT}/testing/data/csv/*.csv ${CORPUS_DIR}
   ^-----------^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean:
cp "${ARROW_ROOT}"/testing/data/csv/*.csv ${CORPUS_DIR}


In generate_corpuses.sh line 89:
${ARROW_CPP}/build-support/fuzzing/pack_corpus.py ${CORPUS_DIR} ${OUT}/arrow-csv-fuzz_seed_corpus.zip
^----------^ SC2086 (info): Double quote to prevent globbing and word splitting.
                                                                ^----^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean:
"${ARROW_CPP}"/build-support/fuzzing/pack_corpus.py ${CORPUS_DIR} "${OUT}"/arrow-csv-fuzz_seed_corpus.zip

For more information:
  https://www.shellcheck.net/wiki/SC2046 -- Quote this to prevent word splitt...
  https://www.shellcheck.net/wiki/SC2128 -- Expanding an array without an ind...
  https://www.shellcheck.net/wiki/SC2086 -- Double quote to prevent globbing ...

What changes are included in this PR?

• SC2046: Quote command substitutions to prevent word splitting.
• SC2128: Specify an array index explicitly.
• SC2086: Quote variables to prevent globbing and word splitting.

Are these changes tested?

Yes.

Are there any user-facing changes?

No.

• GitHub Issue: [CI][C++] Fix shellcheck errors in cpp/build-support/fuzzing/generate_corpuses.sh #51090

[github-actions]

⚠️ GitHub issue #51090 has been automatically assigned in GitHub to PR creator.

[hiroyuki-sato]

@github-actions crossbow submit test-build-cpp-fuzz

[github-actions]

Revision: 534ce6f

Submitted crossbow builds: ursacomputing/crossbow @ actions-08167fab61

Task	Status
test-build-cpp-fuzz

[hiroyuki-sato]

@kou

It looks like generate_corpuses.sh is called when "${ARROW_FUZZING}" == "ON", but I don't know how to enable ARROW_FUZZING.

if [ "${ARROW_FUZZING}" == "ON" ]; then
    # .. snip

    # 1. Generate seed corpuses
    # For IPC fuzz targets, these will include the golden IPC integration files.
    "${source_dir}/build-support/fuzzing/generate_corpuses.sh" "${binary_output_dir}"

ARROW_FUZZING is enabled in the following parts.

cpp/CMakePresets.json
https://github.com/apache/arrow/blob/main/cpp/CMakePresets.json#L465

    {
      "name": "fuzzing",
      "inherits": [
        "sanitizer-asan",
        "sanitizer-ubsan",
        "base"
      ],
      "displayName": "Debug build with IPC and Parquet fuzzing targets",
      "cacheVariables": {
        "ARROW_CSV": "ON",
        "ARROW_DEPENDENCY_SOURCE": "BUNDLED",
        "ARROW_FUZZING": "ON",

ubuntu-cpp-sanitizer
https://github.com/apache/arrow/blob/main/compose.yaml#L685

  ubuntu-cpp-sanitizer:
  #..snip
    environment:
      <<: [*common, *ccache, *sccache, *cpp]
      # snip...
      ARROW_FLIGHT_SQL: "OFF"
      ARROW_FUZZING: "ON"  # Check fuzz regressions  

[kou]

It seems that we have a document for it: https://arrow.apache.org/docs/developers/cpp/fuzzing.html

archery docker run ubuntu-cpp-sanitizer will work too.

[hiroyuki-sato]

@kou Thanks. Does this mean I need to run Archery on my local machine instead of in CI?

[kou]

Yes if you verify it on local.

AMD64 Ubuntu 24.04 C++ ASAN UBSAN also uses it:

https://github.com/apache/arrow/actions/runs/33349187376/job/99359104165?pr=51091#step:8:4543

+ /arrow/cpp/build-support/fuzzing/generate_corpuses.sh /build/cpp/debug

[hiroyuki-sato]

@kou Thanks. Oh, I didn't know that AMD64 Ubuntu 24.04 C++ ASAN UBSAN runs with ARROW_FUZZING=ON.
I'll take a look at the failure logs.
I'll mark this PR as a draft for now.

[hiroyuki-sato]

@kou Could you take a look when you get a chacne? I think the two CI failures are unrelated to this PR.

[hiroyuki-sato]

I forgot that macOS uses Bash 3.2, which doesn't support the mapfile command. I'll rewirte later.

[hiroyuki-sato]

@kou mapfile is not available in Bash 3.2 on macOS, so I simplified the implementation. Please take a look.
The CI failures are unrelated to this PR, and some of them may be fixed by #51012.

[kou]

+1