[GH-2646] Auto-populate covering metadata for GeoParquet 1.1.0 writes by jiayuasu · Pull Request #2665 · apache/sedona

jiayuasu · 2026-02-20T10:07:12Z

Did you read the Contributor Guide?

Yes, I have read the Contributor Rules and Contributor Developer Guide

Is this PR related to a ticket?

Yes, and the PR name follows the format [GH-XXX] my subject. Closes GeoParquet 1.1 covering metadata should auto populate #2646

What changes were proposed in this PR?

Add a geoparquet.covering.mode option to control automatic covering metadata generation when writing GeoParquet files.

Behavior

auto (default): For GeoParquet 1.1.0 writes, automatically generate or reuse <geometryColumnName>_bbox covering columns and write corresponding covering metadata. If the user has already provided explicit geoparquet.covering or geoparquet.covering.<col> options, those take precedence and auto-generation is skipped.
legacy: No automatic covering generation. Explicit covering options still work as before.

Changes

GeoParquetMetaData.scala

Added constants: GEOPARQUET_COVERING_MODE_KEY, GEOPARQUET_COVERING_MODE_AUTO, GEOPARQUET_COVERING_MODE_LEGACY.

GeoParquetWriteSupport.scala

Parse and validate geoparquet.covering.mode from Hadoop configuration. Throw IllegalArgumentException for invalid values.
maybeAutoGenerateCoveringColumns(): when auto mode is enabled and no explicit covering options are provided, for each geometry column: reuse an existing valid _bbox struct column, or generate one from the geometry envelope.
Guard against key collision when a geometry column is named "mode" (skip geoparquet.covering.mode in per-column covering parsing).
Gracefully handle the case where an existing _bbox column has invalid structure (log warning and skip instead of crashing).

geoparquetIOTests.scala

Test auto-covering reuses existing valid geometry_bbox column.
Test auto-covering generates geometry_bbox when no covering column exists.
Test legacy mode disables auto-generation.
Test invalid mode is rejected with a clear error message.
Test auto-covering for multiple geometry columns.
Test auto-covering is not applied for non-1.1.0 versions.
Fix round-trip comparison tests to select only original columns (auto-covering adds geometry_bbox).

geoparquet-sedona-spark.md

Document the geoparquet.covering.mode option, default behavior, and how to opt out.
Note that the default GeoParquet version is 1.1.0 since v1.9.0.

How was this patch tested?

All 40 geoparquetIOTests pass:

mvn test -pl spark/common -Dlog4j.version=2.19.0 -DwildcardSuites=org.apache.sedona.sql.geoparquetIOTests

Did this PR include necessary documentation updates?

Yes, I have updated the documentation.

Copilot

Pull request overview

Adds an opt-in/opt-out mechanism for automatically generating GeoParquet 1.1.0 “covering” metadata (and corresponding <geometry>_bbox columns) during writes, reducing manual setup and making writes safer by default.

Changes:

Introduces geoparquet.covering.mode (auto default, legacy) and validates it during writes.
Implements auto-reuse or auto-generation of <geometryColumnName>_bbox columns + covering metadata for GeoParquet 1.1.0 writes when no explicit covering options are provided.
Updates Scala/R tests and docs to account for auto-added covering columns.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
spark/common/src/main/scala/org/apache/spark/sql/execution/datasources/geoparquet/GeoParquetWriteSupport.scala	Adds covering mode parsing + auto-generation/reuse of `*_bbox` covering columns and metadata during writes.
spark/common/src/main/scala/org/apache/spark/sql/execution/datasources/geoparquet/GeoParquetMetaData.scala	Defines new constants for covering mode option and documents supported values.
spark/common/src/test/scala/org/apache/sedona/sql/geoparquetIOTests.scala	Adds auto-covering tests and adjusts round-trip assertions to ignore auto-generated columns.
docs/tutorial/files/geoparquet-sedona-spark.md	Documents default auto-covering behavior and the `geoparquet.covering.mode` option.
R/tests/testthat/test-data-interface.R	Updates round-trip comparison to select only original columns (excluding auto-generated covering columns).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-20T20:29:54Z

spark/common/src/test/scala/org/apache/sedona/sql/geoparquetIOTests.scala

+    it("GeoParquet auto populates covering metadata for single geometry column") {
+      val df = sparkSession
+        .range(0, 100)
+        .toDF("id")
+        .withColumn("id", expr("CAST(id AS DOUBLE)"))
+        .withColumn("geometry", expr("ST_Point(id, id + 1)"))
+        .withColumn(
+          "geometry_bbox",
+          expr("struct(id AS xmin, id + 1 AS ymin, id AS xmax, id + 1 AS ymax)"))
+      val geoParquetSavePath =


The new behavior to gracefully handle an existing but invalid <geometryColumnName>_bbox struct (log warning + skip) isn’t covered by tests here. Adding a test that writes a DataFrame with a geometry_bbox column missing required fields or with non-float/double types would help ensure the write does not fail and that covering metadata is not populated in that case.

Copilot · 2026-02-20T20:29:55Z

...ain/scala/org/apache/spark/sql/execution/datasources/geoparquet/GeoParquetWriteSupport.scala

+            case _: IllegalArgumentException =>
+              logWarning(
+                s"Existing column '$coveringColumnName' is not a valid covering struct " +
+                  s"(expected struct<xmin, ymin, xmax, ymax> with float/double fields). " +


The warning message here says the expected struct is exactly struct<xmin, ymin, xmax, ymax>, but createCoveringColumnMetadata also supports optional zmin/zmax fields. Consider adjusting the log message to reflect the actual accepted schema (required xmin/ymin/xmax/ymax, optional zmin/zmax) so users aren’t misled when troubleshooting.

Suggested change

s"(expected struct<xmin, ymin, xmax, ymax> with float/double fields). " +

s"(expected struct<xmin, ymin, xmax, ymax> with float/double fields; " +

s"optional zmin/zmax fields are also supported). " +

Copilot · 2026-02-20T20:29:55Z

...ain/scala/org/apache/spark/sql/execution/datasources/geoparquet/GeoParquetWriteSupport.scala

+
+    val generatedCoveringFields = mutable.ArrayBuffer.empty[StructField]
+    val geometryColumns =
+      geometryColumnInfoMap.keys.toSeq.map(ordinal => ordinal -> schema(ordinal).name)


geometryColumnInfoMap is a mutable.Map (hash-based), so iterating geometryColumnInfoMap.keys.toSeq can yield a non-deterministic order. That makes the appended auto-generated <geom>_bbox columns (and their ordinals) potentially vary across runs, which can hurt reproducibility and make downstream expectations flaky. Consider iterating geometry ordinals in schema order (e.g., geometryColumnInfoMap.keys.toSeq.sorted) when building geometryColumns.

Suggested change

geometryColumnInfoMap.keys.toSeq.map(ordinal => ordinal -> schema(ordinal).name)

geometryColumnInfoMap.keys.toSeq.sorted.map(ordinal => ordinal -> schema(ordinal).name)

… test

github-actions bot added docs sedona-common sedona-spark labels Feb 20, 2026

jiayuasu added the API breaking change label Feb 20, 2026

[GH-2646] Auto-populate covering metadata for GeoParquet 1.1.0 writes

5f51568

jiayuasu force-pushed the issue-2646-geoparquet-covering-auto-populate branch from 66f1720 to 5f51568 Compare February 20, 2026 10:08

github-actions bot removed the sedona-common label Feb 20, 2026

jiayuasu added this to the sedona-1.9.0 milestone Feb 20, 2026

Fix R geoparquet round-trip test for auto-covering column

fee77f0

github-actions bot added the sedona-r label Feb 20, 2026

jiayuasu requested a review from Copilot February 20, 2026 20:24

Copilot started reviewing on behalf of jiayuasu February 20, 2026 20:25 View session

Copilot AI reviewed Feb 20, 2026

View reviewed changes

Address review: sort ordinals, fix warning message, add invalid _bbox…

934c5ce

… test

jiayuasu merged commit c5acc9c into master Feb 21, 2026
40 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Comments

[GH-2646] Auto-populate covering metadata for GeoParquet 1.1.0 writes#2665

[GH-2646] Auto-populate covering metadata for GeoParquet 1.1.0 writes#2665
jiayuasu merged 3 commits intomasterfrom
issue-2646-geoparquet-covering-auto-populate

jiayuasu commented Feb 20, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Feb 20, 2026

Uh oh!

Copilot AI Feb 20, 2026

Uh oh!

Copilot AI Feb 20, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

	s"(expected struct<xmin, ymin, xmax, ymax> with float/double fields). " +
	s"(expected struct<xmin, ymin, xmax, ymax> with float/double fields; " +
	s"optional zmin/zmax fields are also supported). " +

	geometryColumnInfoMap.keys.toSeq.map(ordinal => ordinal -> schema(ordinal).name)
	geometryColumnInfoMap.keys.toSeq.sorted.map(ordinal => ordinal -> schema(ordinal).name)

Comments

Conversation

jiayuasu commented Feb 20, 2026

Did you read the Contributor Guide?

Is this PR related to a ticket?

What changes were proposed in this PR?

Behavior

Changes

How was this patch tested?

Did this PR include necessary documentation updates?

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Feb 20, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 20, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 20, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant