GH-34577 [Python] Expose eol and null_string csv WriteOptions

[MartinNowak]

Rationale for this change

Expose remaining csv write options to pyarrow.

What changes are included in this PR?

Adding eol and null_string to csv.WriteOptions.

Are these changes tested?

Yes, testing of setters included.

Are there any user-facing changes?

• GitHub Issue: [R][Python] Expand coverage of and align R/Python to C++ CSV WriteOptions #34577

[github-actions]

⚠️ GitHub issue #34577 has been automatically assigned in GitHub to PR creator.

[AlenkaF]

Thank you for submitting a PR!
There are some fixes needed for the CI to pass:

• Linter error:
  python/pyarrow/tests/test_csv.py:363:89: E501 line too long (119 > 88 characters)
• Python builds error:
  pyarrow/_csv.pyx:1431:34: Object of type 'CCSVWriteOptions &' has no attribute 'eol'
  -> CCSVWriteOptions will need to be updated in the libarrow.pxd

[AlenkaF]

The failures are connected, see:

______________________________ test_write_options ______________________________

    def test_write_options():
        cls = WriteOptions
        opts = cls()
    
>       check_options_class(
            cls, include_header=[True, False], delimiter=[',', '\t', '|'],
            eol=['\n', '\r\n'], null_string=['', 'NA'],
            quoting_style=['needed', 'none', 'all_valid'])

opt/conda/envs/arrow/lib/python3.9/site-packages/pyarrow/tests/test_csv.py:362: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
opt/conda/envs/arrow/lib/python3.9/site-packages/pyarrow/tests/test_csv.py:111: in check_options_class
    opts = cls(**non_defaults)
pyarrow/_csv.pyx:1388: in pyarrow._csv.WriteOptions.__init__
    ???
pyarrow/_csv.pyx:1446: in pyarrow._csv.WriteOptions.null_string.__set__
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   TypeError: expected bytes, NoneType found

[MartinNowak]

Was on vacation and it took a bit longer to get back to this. Hope it runs through now 🤞 @AlenkaF.

[MartinNowak]

???
E TypeError: expected bytes, NoneType found

It found this glorious typo 🤦. Only have semi-working syntax highlighting and didn't have enough time to figure out how to run the tests locally :/.

[AlenkaF]

The tests are passing 👍
But the docstrings need an update:

pyarrow._csv.WriteOptions
-> pyarrow._csv.WriteOptions(include_header=None, *, batch_size=None, delimiter=None, eol=None, null_string=None, quoting_style=None)
PR01: Parameters {'delimiter', 'null_string', 'eol', 'batch_size', 'quoting_style'} not documented
PR04: Parameter "")" has no type

[MartinNowak]

The tests are passing 👍
But the docstrings need an update:

Needed to escape the backslash to render in final doc rather than to break the doc comment.

[MartinNowak]

The macOS failures seem to be apache/orc#2357 and were not present previously f95d305 @AlenkaF.

[raulcd]

The macOS failures are fixed on main, can you rebase so they are fixed on CI, please?

[raulcd]

@github-actions crossbow submit -g python

[github-actions]

Revision: 08fa4412afc3bbb2e1599a4ad9557fe8a0f3a075

Submitted crossbow builds: ursacomputing/crossbow @ actions-c05e9be928

Task	Status
example-python-minimal-build-fedora-conda
example-python-minimal-build-ubuntu-venv
test-conda-python-3.10
test-conda-python-3.10-hdfs-2.9.2
test-conda-python-3.10-hdfs-3.2.1
test-conda-python-3.10-pandas-latest-numpy-latest
test-conda-python-3.11
test-conda-python-3.11-dask-latest
test-conda-python-3.11-dask-upstream_devel
test-conda-python-3.11-hypothesis
test-conda-python-3.11-pandas-latest-numpy-1.26
test-conda-python-3.11-pandas-latest-numpy-latest
test-conda-python-3.11-pandas-nightly-numpy-nightly
test-conda-python-3.11-pandas-upstream_devel-numpy-nightly
test-conda-python-3.11-spark-master
test-conda-python-3.12
test-conda-python-3.12-cpython-debug
test-conda-python-3.13
test-conda-python-3.9
test-conda-python-3.9-pandas-1.1.3-numpy-1.19.5
test-conda-python-emscripten
test-cuda-python-ubuntu-22.04-cuda-11.7.1
test-debian-12-python-3-amd64
test-debian-12-python-3-i386
test-fedora-42-python-3
test-ubuntu-22.04-python-3
test-ubuntu-22.04-python-313-freethreading
test-ubuntu-24.04-python-3

[raulcd]

Thanks for the PR, sorry I took a little to review.
This seems to only be testing that the options class can be generated but is not indeed testing the options, on test_csv.py we have other tests that indeed test the options like:

arrow/python/pyarrow/tests/test_csv.py

Line 1929 in 697f501

def test_write_read_round_trip():

or

arrow/python/pyarrow/tests/test_csv.py

Line 1973 in 697f501

def test_write_quoting_style():

Can we test the options are indeed working as expected when writing the CSV?

[github-actions]

Thank you for your contribution. Unfortunately, this pull request has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this PR will be closed in 14 days. Feel free to re-open this if it has been closed in error. If you do not have repository permissions to reopen the PR, please tag a maintainer.

[AlenkaF]

I pushed the tests so the PR does not go stale. @raulcd are you OK with me merging if the CI is green?

[AlenkaF]

@github-actions crossbow submit -g python

[github-actions]

Revision: 6466887

Submitted crossbow builds: ursacomputing/crossbow @ actions-33214ccef0

Task	Status
example-python-minimal-build-fedora-conda
example-python-minimal-build-ubuntu-venv
test-conda-python-3.11
test-conda-python-3.11-dask-latest
test-conda-python-3.11-dask-upstream_devel
test-conda-python-3.11-hdfs-2.9.2
test-conda-python-3.11-hdfs-3.2.1
test-conda-python-3.11-hypothesis
test-conda-python-3.11-pandas-2.2.2-numpy-2.0.2
test-conda-python-3.11-spark-master
test-conda-python-3.12
test-conda-python-3.12-pandas-latest-numpy-latest
test-conda-python-3.13
test-conda-python-3.13-pandas-latest-numpy-2.1.3
test-conda-python-3.13-pandas-latest-numpy-latest
test-conda-python-3.14
test-conda-python-3.14-cpython-debug
test-conda-python-3.14-pandas-nightly-numpy-nightly
test-conda-python-3.14-pandas-upstream_devel-numpy-nightly
test-conda-python-emscripten
test-debian-13-python-3-amd64
test-debian-13-python-3-i386
test-fedora-42-python-3
test-ubuntu-24.04-python-3

[raulcd]

@raulcd are you OK with me merging if the CI is green?

Sure, the current linter error seems related.

[AlenkaF]

CI is green, example-* extended build failures are not related.
Will merge.