GH-47686: [Docs][Python] Split the Python Parquet docs into separate items

[AlenkaF]

Rationale for this change

The current parquet section is one-page and covering lots of topics. As per #46601 the parquet section is most visited and would therefore need a bit of maintenance/content update.

What changes are included in this PR?

• Parquet documentation is split in 4 different subpages (index.rst, parquet.rst, parquet_datasets.rst and parquet_encryption.rst).
• Some additional content is added

Are these changes tested?

Via the docs preview

Are there any user-facing changes?

No

• GitHub Issue: [Docs][Python] Split the Python Parquet docs into separate items #47686

[AlenkaF]

@github-actions crossbow submit preview-docs

[github-actions]

Revision: 156850c

Submitted crossbow builds: ursacomputing/crossbow @ actions-937dad9db2

Task	Status
preview-docs

[AlenkaF]

I have used Claude for suggesting the split of pyarrow parquet page and to look through the code and the git log history for any features/changes that might be missing from the docs. From what was suggested I continued with the work on my own.

I have split the changes in different commits so the change will be easier to review (due to the docs split, the change is bigger than it really is).

commit 1: restructure of the content, no content changes @thisisnic, do you mind having a look?
commit 2: add content about filtering and ability to write bloom filters @mapleFU, do you mind having a look?
commit 3: add section on iter_batches @raulcd, do you mind having a look?
commit 4: add section on binary/list_type option on read, reading extension types @pitrou @paleolimbot, do you mind having a look?
commit 5: datasets note on single file partitions not being included
commit 6: update parquet writer's parameter list @raulcd, do you mind having a look?
commit 7: update encryption section adding info on key rotation and direct encryption @adamreeve, do you mind having a look?

Rendered docs are available here: https://s3.amazonaws.com/arrow-data/pr_docs/50419/python/parquet/index.html

[Copilot]

Pull request overview

Splits the PyArrow Parquet user-guide documentation into multiple focused pages to improve navigability and enable incremental maintenance, aligning with the broader Python docs revision effort.

Changes:

• Replaces the single large Parquet guide page with a section index plus dedicated subpages (single-file IO, datasets, type handling, encryption).
• Updates the Python docs toctree to point at the new Parquet section entry point.
• Adds/relocates some additional Parquet-related content during the split (e.g., content-defined chunking).

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
docs/source/python/parquet/parquet.rst	New “single files” Parquet IO page (read/write, metadata, filtering, etc.).
docs/source/python/parquet/parquet_type_handling.rst	New page for Parquet read/write type-handling options.
docs/source/python/parquet/parquet_datasets.rst	New page for partitioned datasets / multi-file Parquet usage.
docs/source/python/parquet/parquet_encryption.rst	New page for Parquet modular encryption docs.
docs/source/python/parquet/index.rst	New Parquet section landing page with toctrees to subpages.
docs/source/python/parquet.rst	Removes the previous monolithic Parquet documentation page.
docs/source/python/index.rst	Updates the Python guide toctree to reference the new Parquet section entry.

[paleolimbot]

Thank you!

I took a read through and this content seems well organized and I appreciate the additions for Geometry/Geography types!

[paleolimbot]

🎉 !

[adamreeve]

Thanks @AlenkaF, I've skimmed over most of this and looked more thoroughly at the encryption section changes.

[AlenkaF]

Thank you all for the reviews!
I still need to go through the Copilot review comments, but I have applied other review comment suggestions.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.

Comments suppressed due to low confidence (3)

docs/source/python/parquet/parquet_type_handling.rst:120

• Sentence casing/grammar: “In addition, We provide …” should be “In addition, we provide …”.

In addition, We provide the ``coerce_timestamps`` option to allow you to select

docs/source/python/parquet/parquet_encryption.rst:71

• Grammar: “as following” should be “as follows”.

defined by :class:`pyarrow.parquet.encryption.KmsClient` as following:

docs/source/python/parquet/parquet_datasets.rst:26

• Grammar: “These may present in a number of ways” is missing “be”.

Multiple Parquet files constitute a Parquet *dataset*. These may present in a
number of ways:

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.

Comments suppressed due to low confidence (4)

docs/source/python/parquet/parquet_encryption.rst:108

• :download: target is now one directory deeper (docs/source/python/parquet/…), so ../../../python/... resolves under docs/ and breaks the download link. Update the relative path to reach repo root.

An :download:`example <../../../../python/examples/parquet_encryption/sample_vault_kms_client.py>`

docs/source/python/parquet/parquet_type_handling.rst:73

• This code example uses pq.write_table / pq.read_table but doesn't import pyarrow.parquet as pq, making the snippet non-runnable as written.

   >>> table = pa.table({'data': pa.array([b'hello', b'world'], pa.binary())})
   >>> pq.write_table(table, 'binary.parquet', store_schema=False)
   >>> pq.read_table('binary.parquet', binary_type=pa.large_binary()).schema
   data: large_binary

docs/source/python/parquet/parquet_type_handling.rst:121

• Sentence casing/grammar: "In addition, We provide" should use lowercase "we".

the desired resolution:

docs/source/python/parquet/parquet_datasets.rst:26

• Grammar: "These may present in a number of ways" is missing a verb (should be "may be present" / "may be presented").

Multiple Parquet files constitute a Parquet *dataset*. These may be present in a
number of ways:

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

docs/source/python/parquet/parquet.rst:284

• The cross-reference uses :func:`~parquet.read_metadata``` which likely resolves to a top-level parquetmodule rather thanpyarrow.parquet`` and can break the generated link. Use a relative target under the currentmodule (or the fully-qualified name).

or can also be read directly using :func:`~parquet.read_metadata`:

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (2)

docs/source/python/parquet/parquet.rst:269

• :class:~.ParquetFile`` resolves to pyarrow.ParquetFile under `.. currentmodule:: pyarrow`, but the actual class is `pyarrow.parquet.ParquetFile`. Use the fully-qualified reference so Sphinx can link it correctly.

The ``FileMetaData`` of a Parquet file can be accessed through
:class:`~.ParquetFile` as shown above:

docs/source/python/index.rst:57

• Switching the root toctree entry from parquet to parquet/index removes the generated python/parquet.html page (the old docs/source/python/parquet.rst is deleted), which will break existing inbound links/bookmarks. Consider keeping docs/source/python/parquet.rst as a shim landing page (old URL) that links/includes the new subpages, and keep the toctree entry as parquet.

   timestamps
   parquet/index
   dataset

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

docs/source/python/parquet/parquet.rst:103

• This section labels the output as "RSS" but prints pa.total_allocated_bytes() (Arrow-allocated bytes), and it also uses path without defining it. This makes the example misleading and not directly copy/pasteable.

Because Parquet data needs to be decoded from the Parquet format
and compression, it can't be directly mapped from disk.
Thus the ``memory_map`` option might perform better on some systems
but won't help much with resident memory consumption.

docs/source/python/parquet/index.rst:72

• PR description says the Parquet docs are split into 4 subpages, but the docs index includes an additional subpage (parquet_type_handling). Please update the PR description (or adjust the split) so the change summary matches what is actually being added.

Data Type Handling
------------------

.. toctree::
   :maxdepth: 2

   parquet_type_handling

[AlenkaF]

@github-actions crossbow submit preview-docs

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (4)

docs/source/python/index.rst:57

• The PR description says the docs are split into 4 subpages, but this diff also adds parquet_type_handling.rst (so at least 5 pages including the new index). Also, switching from docs/source/python/parquet.rst to docs/source/python/parquet/index.rst changes the built URL from python/parquet.html to python/parquet/index.html and can break inbound links; consider keeping a stub python/parquet.rst page that links/redirects to the new index to preserve the old URL, and update the PR description accordingly.

   timestamps
   parquet/index
   dataset

docs/source/python/parquet/parquet.rst:248

• :meth: role is relative to .. currentmodule:: pyarrow, so :meth:~.ParquetFile.iter_batches`` will try to resolve pyarrow.ParquetFile.iter_batches (which doesn’t exist). Use the fully-qualified Parquet class path so Sphinx can resolve the method.

For memory-efficient reads of large files, :meth:`~.ParquetFile.iter_batches`
streams the file as a sequence of :class:`~pyarrow.RecordBatch` objects rather

docs/source/python/parquet/parquet.rst:270

• This reference is relative to .. currentmodule:: pyarrow, so :class:~.ParquetFile`` resolves to pyarrow.ParquetFile (not `pyarrow.parquet.ParquetFile`) and is likely to be an unresolved reference in the rendered docs. Use the fully-qualified class name (consistent with earlier in this page).

The ``FileMetaData`` of a Parquet file can be accessed through
:class:`~.ParquetFile` as shown above:

docs/source/python/parquet/parquet.rst:285

• With .. currentmodule:: pyarrow, :func:~parquet.read_metadata`` points at a top-level parquet module, not `pyarrow.parquet`. This is likely to be an unresolved reference; use `pyarrow.parquet.read_metadata` (or `.parquet.read_metadata`).

or can also be read directly using :func:`~parquet.read_metadata`:

[AlenkaF]

@github-actions crossbow submit preview-docs

[github-actions]

Revision: 706fa29

Submitted crossbow builds: ursacomputing/crossbow @ actions-44053c8153

Task	Status
preview-docs

[AlenkaF]

@github-actions crossbow submit preview-docs

[github-actions]

Revision: 706fa29

Submitted crossbow builds: ursacomputing/crossbow @ actions-20af5b3716

Task	Status
preview-docs

[AlenkaF]

Rendered docs available here: https://s3.amazonaws.com/arrow-data/pr_docs/50419/python/parquet/index.html
I will keep this PR open for a week before merging, in case there are any more comments.

[AlenkaF]

@thisisnic would you have some time to look through the general splitting of the Parquet section in Python User Guide? This change has been done in commit 1. The preview can be found on the following link: https://s3.amazonaws.com/arrow-data/pr_docs/50419/python/parquet/index.html

[AlenkaF]

Will merge. Any comments can be included as a follow-up.