GH-47499: [Python] explicitly add use_content_defined_chunking as parameter to ParquetWriter and ParquetWriter.write_table and show it as a parameter in doc page

[severo]

Rationale for this change

The doc page https://arrow.apache.org/docs/python/generated/pyarrow.parquet.write_table.html does not show use_content_defined_chunking as an argument of write_table:

Note that it's documented below in the list of parameters:

What changes are included in this PR?

Change the API signature of ParquetWriter and ParquetWriter.write_table: use_content_defined_chunking parameter is explicitly added, with a default value (False), as mentioned by the comment:

arrow/python/pyarrow/parquet/core.py

Lines 1953 to 1954 in 512dcd7

	# Implementor's note: when adding keywords here / updating defaults, also
	# update it in write_to_dataset and _dataset_parquet.pyx ParquetFileWriteOptions

Reorder the parameters in docstring _parquet_writer_arg_docs to match the method signature.

Add a test.

Also users won't get this error anymore TypeError: unexpected parquet write option: use_content_defined_chunking as it happens now when running with released wheel since 21.0.0 :

python -c "import pyarrow.dataset as ds; ds.ParquetFileFormat().make_write_options(use_content_defined_chunking=True)"

or when running :

import pyarrow as pa, pyarrow.parquet as pq
pq.write_to_dataset(pa.table({'a':[1, 2, 3]}), 'cdc_dataset', use_content_defined_chunking=True)

Are these changes tested?

Yes

Are there any user-facing changes?

Yes:

1. use_content_defined_chunking parameter is explicitly added to the API signature of ParquetWriter and ParquetWriter.write_table
2. the docs now show use_content_defined_chunking as an argument of write_table.

• GitHub Issue: [Python] Show use_content_defined_chunking in the list of parameters of parquet.write_table (doc) #47499

[github-actions]

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

See also:

• Other pull requests
• Contribution Guidelines - Contributing Overview

[severo]

cc @kszucs

[github-actions]

⚠️ GitHub issue #47499 has been automatically assigned in GitHub to PR creator.

[severo]

Also fixes #47477

[github-actions]

Thank you for your contribution. Unfortunately, this pull request has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this PR will be closed in 14 days. Feel free to re-open this if it has been closed in error. If you do not have repository permissions to reopen the PR, please tag a maintainer.

[tadeja]

@severo thank you for your past contribution here! Would you be available to continue with this pull request, to rebase onto the latest main? And perhaps add an extra test like;

diff --git a/python/pyarrow/tests/test_dataset.py b/python/pyarrow/tests/test_dataset.py
--- a/python/pyarrow/tests/test_dataset.py
+++ b/python/pyarrow/tests/test_dataset.py
@@ -5930,6 +5930,19 @@ def test_checksum_write_dataset_read_dataset_to_table(tempdir):
         ).to_table()
 
 
+@pytest.mark.parquet
+@pytest.mark.parametrize("cdc", [
+    True,
+    {"min_chunk_size": 32 * 1024, "max_chunk_size": 64 * 1024},
+])
+def test_write_dataset_content_defined_chunking(tempdir, cdc):
+    expected_table = pa.table({'a': [1, 2, 3]})
+    fmt = ds.ParquetFileFormat()
+    opts = fmt.make_write_options(use_content_defined_chunking=cdc)
+    ds.write_dataset(expected_table, tempdir, format=fmt, file_options=opts)
+    assert ds.dataset(tempdir, format=fmt).to_table().equals(expected_table)
+
+

[tadeja]

Regarding #47477's side request

Also, it will be useful to have a note about the consequence of setting use_content_defined_chunking=True on systems that do not support CDC.

• What about adding a note in this PR to /docs/source/python/parquet/parquet.rst and/or /python/pyarrow/parquet/core.py ;
  "On storage that does not deduplicate content (non-CAS) the only effect may be more and smaller data pages and a potentially larger file." < - would this be OK, @kszucs ?

[kszucs]

@tadeja essentially there are no meaningful effects on non-CAS storage systems, the data pages won't be fixed sized but mostly that's it.

[severo]

Done. Thanks for the test.

[Copilot]

🟢 Approval recommended

The change is a straightforward signature/plumbing update with a targeted regression test and consistent wiring to existing underlying option handling.

Pull request overview

Expose use_content_defined_chunking in the public Python API signatures (notably pyarrow.parquet.write_table) so it appears in the generated documentation parameter list, while keeping the existing behavior and wiring through to the underlying Parquet writer properties.

Changes:

• Added use_content_defined_chunking (default False) to ParquetWriter.__init__ and write_table signatures and forwarded it into the underlying writer.
• Plumbed use_content_defined_chunking into dataset Parquet write options (ParquetFileWriteOptions) so ParquetFileFormat.make_write_options(use_content_defined_chunking=...) works.
• Added a dataset-level regression test covering use_content_defined_chunking for both boolean and dict configurations.

File summaries

File	Description
python/pyarrow/parquet/core.py	Adds the missing keyword to public signatures and forwards it to the underlying Parquet writer so docs show it.
python/pyarrow/_dataset_parquet.pyx	Adds the option to dataset Parquet write options defaults and passes it into writer property creation.
python/pyarrow/tests/test_dataset.py	Adds coverage for dataset writes using content-defined chunking (bool and dict forms).

Review details

• Files reviewed: 3/3 changed files
• Comments generated: 0
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🟡 Changes recommended

The write_table signature change reorders/inserts positional-or-keyword parameters, which is a backward-incompatible API change for positional callers.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 3/3 changed files
• Comments generated: 1
• Review effort level: Lite

[severo]

OK, reverted and fixed

[Copilot]

🟢 Approval recommended

The changes consistently expose and plumb the option through writer and dataset paths, and include a targeted regression test for the new dataset behavior.

Review details

• Files reviewed: 3/3 changed files
• Comments generated: 0 new
• Review effort level: Lite

[tadeja]

Thanks @severo, also for fixing numpydoc validation PR03 order error that finally showed up here in the CI Numpydoc job https://github.com/apache/arrow/actions/runs/34203717165/job/101988921363#step:6:4415

pyarrow.parquet.core.write_table
PR03: Wrong parameters order.
Actual: ('table', ..., 'data_page_size', 'flavor', ..., 'write_time_adjusted_to_utc', 'max_rows_per_page', 'bloom_filter_options', 'use_content_defined_chunking', '**kwargs').
Documented: ('table', ..., 'data_page_size', 'max_rows_per_page', 'flavor', ..., 'use_content_defined_chunking', 'write_time_adjusted_to_utc', 'bloom_filter_options', '**kwargs')

• Apparently max_rows_per_page is out of order on main for a while, but numpydoc validation skips PR03 order check if a documented parameter is missing (= PR02 check is still disabled on Arrow). Because use_content_defined_chunking was present in docstring but missing from the signature, the order check PR03 never ran for write_table!
  -> EDIT: I've just opened a separate issue [Python][Docs][CI] Disabled numpydoc PR02 check may skip PR03 #51260

[tadeja]

@severo Could you just update PR description with the newest changes?
Also users won't get this error anymore
TypeError: unexpected parquet write option: use_content_defined_chunking
as it happens now when running with released wheel since 21.0.0 :

python -c "import pyarrow.dataset as ds; ds.ParquetFileFormat().make_write_options(use_content_defined_chunking=True)"

or when running :

import pyarrow as pa, pyarrow.parquet as pq
pq.write_to_dataset(pa.table({'a':[1, 2, 3]}), 'cdc_dataset', use_content_defined_chunking=True)

[severo]

Could you just update PR description with the newest changes?

Done. Feel free to update it directly if you think something is missing or wrong.

[tadeja]

@github-actions crossbow submit preview-docs

[github-actions]

Revision: 7e35b0f

Submitted crossbow builds: ursacomputing/crossbow @ actions-020ac4b3c4

Task	Status
preview-docs

[tadeja]

@github-actions crossbow submit preview-docs

[github-actions]

Revision: 0d5d7b7

Submitted crossbow builds: ursacomputing/crossbow @ actions-d1c4d05d2b

Task	Status
preview-docs

[Copilot]

🟡 Changes recommended

The dataset test must use sufficiently large data and observable CDC-specific output to validate option propagation.

Get a fresh assessment by requesting another Copilot review.

Review details

• Files reviewed: 3/3 changed files
• Comments generated: 1
• Review effort level: Lite

[tadeja]

I've just done a fresh rebase to get around the failing preview-docs (minIO error 410 gone, resolved by recent commit 7e3b55a).
I would merge this once new CI jobs and preview-docs show up OK. Perhaps any comments, @rok ?

[tadeja]

Generated docs preview pages : pyarrow.parquet.write_table and pyarrow.parquet.ParquetWriter

[rok]

Thanks for working on this @severo ! I think this PR is ready to merge. One nit - could you update PR/issue text/title to reflect that this also technically changes the API signature? (use_content_defined_chunking parameter is explicitly added to ParquetWriter and ParquetWriter.write_table). Also see #47498 (review) regarding the latent dataset bug this fixes.

[severo]

Done!

[rok]

Thanks!