Parquet: Validate geospatial projection parameters - #17578
Conversation
Generated-by: Codex
There was a problem hiding this comment.
Pull request overview
Validates Parquet geospatial annotations against projected Iceberg types.
Changes:
- Reuses primitive-type conversion for validation.
- Rejects CRS and edge-algorithm mismatches.
- Adds focused geospatial projection tests.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
PruneColumns.java |
Validates projected geospatial parameters. |
MessageTypeToType.java |
Extracts reusable primitive conversion. |
TestPruneColumns.java |
Tests matching and mismatched parameters. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Generated-by: Codex
Generated-by: Codex
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.
Suppressed comments (1)
parquet/src/main/java/org/apache/iceberg/parquet/ParquetSchemaUtil.java:165
- This validates only top-level primitives in the ID-less fallback path. When a selected top-level field is a struct, list, or map, the entire Parquet group is retained unchanged, so any projected nested geometry/geography still bypasses the new CRS/edge-algorithm check and can be read with incompatible parameters. Please traverse each matched group positionally against the projected type and validate its nested primitives as well; add a nested ID-less mismatch test to cover this path.
Types.NestedField expectedField = expectedSchema.findField(ordinal);
if (type.isPrimitive() && expectedField.type().isPrimitiveType()) {
PruneColumns.validatePrimitive(
expectedField.type().asPrimitiveType(), type.asPrimitiveType());
|
@huan233usc Please take a look, thanks! |
| .named("table"); | ||
| Schema projection = new Schema(NestedField.optional(1, "geog", GeographyType.crs84())); | ||
|
|
||
| assertThatThrownBy(() -> ParquetSchemaUtil.pruneColumns(fileSchema, projection)) |
There was a problem hiding this comment.
From what I understood, the exception is thrown because the algorithm is different (ie EdgeInterPolationAlgorithm.KARNEY). So shouldn't we be looking for that in the error string?
anoopj
left a comment
There was a problem hiding this comment.
Code change LGTM. Just one comment in the test.
Generated-by: Codex
Summary
Context
Parquet geospatial values carry their CRS and edge interpolation algorithm in the logical-type annotation rather than in the WKB payload.
PruneColumnspreviously ignored these parameters, so a file could be interpreted using incompatible parameters from the projected Iceberg schema.This is the Java-side counterpart discovered while reviewing apache/iceberg-cpp#880.
Testing
./gradlew :iceberg-parquet:test --tests org.apache.iceberg.parquet.TestPruneColumns./gradlew :iceberg-parquet:spotlessJavaCheckgit diff --checkAI Disclosure