fix(manifest): write null instead of empty list/map for unset optional DataFile fields - #934
Open
Angelia-Wang wants to merge 1 commit into
Open
Conversation
…l DataFile fields AppendIntList, AppendIntMap, and AppendBinaryMap in src/iceberg/arrow_row_builder.cc always call ArrowArrayFinishElement() after appending entries, even when the input container is empty. This produced an empty-but-non-null list/map element rather than a null element whenever one of DataFile's optional list/map fields (split_offsets, equality_ids, column_sizes, value_counts, null_value_counts, nan_value_counts, lower_bounds, upper_bounds) was unset. Add an empty check to each of these helpers that delegates to the existing AppendNull() instead of finishing an empty element. AppendStringMap (used for required properties-style maps) is intentionally left unchanged. Add unit test coverage in arrow_row_builder_test.cc asserting that AppendIntList/AppendIntMap/AppendBinaryMap write a null element (not an empty one) for empty input, alongside existing non-empty-input coverage.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
AppendIntList,AppendIntMap, andAppendBinaryMapinsrc/iceberg/arrow_row_builder.ccalways callArrowArrayFinishElement()after appending entries, even when the input container is empty. As a result, whenever one of DataFile's optional list/map fields is unset, the C++ writer encodes an empty-but-non-null list/map element in the Avro manifest instead of a null element.These three helpers back all of DataFile's optional list/map fields in
manifest_adapter.cc:split_offsets,equality_ids(optional list)column_sizes,value_counts,null_value_counts,nan_value_counts,lower_bounds,upper_bounds(optional map)Why
The Java reference implementation treats "unset" as null for these same fields:
BaseFileinitializescolumnSizes,valueCounts,nullValueCounts,nanValueCounts,lowerBounds,upperBounds,splitOffsets, andequalityIdstonull(not empty containers), and itsget(pos, ...)accessor — used by the Avro manifest writer — returnsnullfor an unset field (https://github.com/apache/iceberg/blob/1.1.x/core/src/main/java/org/apache/iceberg/BaseFile.java).So manifests written by iceberg-cpp currently diverge from manifests written by iceberg-java whenever any of these 8 fields is unset.
This divergence is not cosmetic — it causes silent data loss when the table is read back with iceberg-java. We verified this end-to-end: after iceberg-cpp commits a snapshot, reading the table with
IcebergGenerics.read(table)(iceberg 1.1.0) returns 0 rows instead of the committed rows:split_offsetsunset → encoded as empty Avro array[]instead ofnull.BaseFile.splitOffsets()converts the empty array into a non-null emptyList<Long>(https://github.com/apache/iceberg/blob/1.1.x/core/src/main/java/org/apache/iceberg/BaseFile.java).BaseContentScanTask.split()selectsOffsetsAwareSplitScanTaskIterator, becausefile.splitOffsets() != null && OFFSET_ORDERING.isOrdered(...)holds for an empty list (https://github.com/apache/iceberg/blob/1.1.x/core/src/main/java/org/apache/iceberg/BaseContentScanTask.java).OffsetsAwareSplitScanTaskIteratorproduces zero split tasks for an empty offset list:splitSizesstays empty andhasNext()always returns false (https://github.com/apache/iceberg/blob/1.1.x/core/src/main/java/org/apache/iceberg/OffsetsAwareSplitScanTaskIterator.java).With this change (null instead of empty list), the same end-to-end test reads back the expected rows: a
nullsplit_offsetsmakesBaseContentScanTask.split()fall back toFixedSizeSplitScanTaskIteratorand the file is read normally.How
Add an empty check to each of the three helpers that delegates to the existing
AppendNull()instead of finishing an empty element. This covers all 8 optional list/map fields with one consistent change;manifest_adapter.ccis untouched.AppendStringMap(used for required properties-style maps) is intentionally left unchanged: an empty-but-non-null map is the expected encoding for required maps (covered by the existingArrowRowBuilderTest.BuildsRowsWithTypedValues).Testing
arrow_row_builder_test.ccassert thatAppendIntList/AppendIntMap/AppendBinaryMapwrite a null element (not an empty one) for empty input, alongside existing non-empty coverage.IcebergGenerics.readreader): without the fix, 0 rows are read back; with the fix, all committed rows are read back.