[Python][Parquet] Support reading and writing Parquet Variant columns

[qzyu999]

Describe the enhancement requested

Description:

The C++ schema mapping between Parquet's VARIANT logical type and Arrow's VariantExtensionType (arrow.parquet.variant) is established by GH-46104. With the C++ Variant encoder (GH-45947) and decoder (GH-45946) implementations, the underlying C++ Parquet reader/writer will be able to process actual data payloads.

This issue tracks Python-level Parquet integration and testing to ensure Python users can read and write Parquet files containing Variant columns seamlessly.

Proposed Changes:

1. Parquet Read/Write Pipeline Validation:
   • Ensure that pyarrow.parquet.write_table correctly serializes VariantExtensionType columns into Parquet files with the VARIANT logical type annotation.
   • Ensure that pyarrow.parquet.read_table correctly deserializes Parquet VARIANT columns back into PyArrow VariantArray columns (rather than falling back to the raw binary-pair storage struct or throwing an unsupported type exception).
2. Metadata and Schema Inspection:
   • Verify that pyarrow.parquet.read_schema and ParquetFile.schema correctly report the column type as the VariantExtensionType extension type.
3. Integration Testing:
   • Add end-to-end tests in python/pyarrow/tests/parquet/test_data_types.py (or a dedicated integration test suite):
     • Construct a pyarrow.Table containing a VariantArray (e.g., from nested dictionaries/lists).
     • Write it to a file using pyarrow.parquet.write_table.
     • Read the file back using pyarrow.parquet.read_table and assert that the types and values are identical to the original table.
     • Test reading a reference Parquet file containing Variant data written by a different implementation (e.g., Go or Spark) to verify cross-language compatibility.

Dependencies:

This issue is blocked by:

• [Python] Bindings for Variant canonical extension type #50131: [Python] Bindings for Variant canonical extension type (Exposing the Python VariantType, VariantArray, and VariantScalar classes)
• GH-45937: [C++][Parquet] Variant decoding #45946 (PR GH-45946: [C++][Parquet] Variant decoding #50121): [C++][Parquet] Variant decoding
• GH-45937: [C++][Parquet] Variant encoding #45947 (PR GH-45947 : [C++][Parquet] Variant encoding #50122): [C++][Parquet] Variant encoding

Component(s)

Python, Parquet

[qzyu999]

take