Skip to content

HIVE-30127: Iceberg: Upgrade iceberg version to 1.12.0 - #6856

Open
Aggarwal-Raghav wants to merge 7 commits into
apache:masterfrom
Aggarwal-Raghav:HIVE-30127
Open

HIVE-30127: Iceberg: Upgrade iceberg version to 1.12.0#6856
Aggarwal-Raghav wants to merge 7 commits into
apache:masterfrom
Aggarwal-Raghav:HIVE-30127

Conversation

@Aggarwal-Raghav

@Aggarwal-Raghav Aggarwal-Raghav commented Oct 7, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Commits made in upsteam iceberg b/w 1.11.0 and 1.12.0 in hive-metastore module:

❯ git log apache-iceberg-1.11.0..apache-iceberg-1.12.0 --oneline -- hive-metastore
e74344ca56 Core, Hive: Commit manifest list encryption keys with the snapshot that uses them (#17984)
0630297ef2 Core, Hive: Fix NPE in commit status check when a create commit fails (#17464)
20d616d6a1 Core, Hive, Spark: Auto close JsonGenerator using try-with-resources (#17931)
4a1513f1f2 API, AWS, BigQuery, Core, Flink, Hive, Nessie, ORC, Parquet, Spark: Use Locale.ROOT instead of Locale.ENGLISH for case conversion (#17442)
787076dc0c Hive: Fix integer overflow in HMS createTime and lastAccessTime (#16620)
8f4843394c Hive: Use server-side filter to list Iceberg tables in HiveCatalog (#17317)
0ddbeead67 Core, AWS, GCP, Dell, Hive: Fix FileIO leaks and standardize close() across Catalog implementations (#16862)
687c58fc7c Core: v4 table metadata location should be optional (#16572)

Important points:

  1. FileWriterFactory Refactor: Upstream Iceberg 1.12.0 deleted the BaseFileWriterFactory and GenericAppenderFactory classes. To adapt, HiveFileWriterFactory was refactored Data: Remove deprecated GenericAppenderFactory and BaseFileWriterFactory iceberg#17696
  2. Positional Delete API Updates: Iceberg 1.12.0 removed the .row() getter and .set(path, pos, row) setter from PositionDelete to prepare for V3 Puffin Deletion Vectors. Core, Data, Spark, Flink: Remove of position delete files with row data iceberg#17706
  3. Updated HMSCatalogAdapter.java to parse and pass the new SnapshotMode (ALL, REFS, NONE) into CatalogHandlers.loadTable
  4. Removed positionDeleteRowSchema after refactor
  5. Added apache5-client dependency as there was one UT failure (iceberg_rest_catalog_gravitino.q) Build: Bump AWS SDK to 2.54.17 and migrate to Apache HttpClient 5 iceberg#18195
 ... 15 more
Caused by: java.lang.NoClassDefFoundError: software/amazon/awssdk/http/apache5/Apache5HttpClient$Builder
 at java.base/java.lang.Class.getDeclaredMethods0(Native Method)
 at java.base/java.lang.Class.privateGetDeclaredMethods(Class.java:3578)
 at java.base/java.lang.Class.getDeclaredMethod(Class.java:2846)
 at org.apache.iceberg.common.DynMethods$Builder.hiddenImpl(DynMethods.java:432)
 at org.apache.iceberg.common.DynMethods$Builder.hiddenImpl(DynMethods.java:391)
 at org.apache.iceberg.common.DynMethods$Builder.hiddenImpl(DynMethods.java:410)
 at org.apache.iceberg.aws.HttpClientProperties.loadHttpClientConfigurations(HttpClientProperties.java:270)
 at org.apache.iceberg.aws.HttpClientProperties.applyHttpClientConfigurations(HttpClientProperties.java:251)
 at software.amazon.awssdk.utils.builder.SdkBuilder.applyMutation(SdkBuilder.java:61)
 at org.apache.iceberg.aws.AwsClientFactories$DefaultAwsClientFactory.s3(AwsClientFactories.java:112)
 at org.apache.iceberg.aws.s3.PrefixedS3Client.s3(PrefixedS3Client.java:85)

Why are the changes needed?

Upgrade Iceberg to 1.12.0

Does this PR introduce any user-facing change?

No

How was this patch tested?

Will see CI outcome, Build compilation ran fine and ran few Iceberg q files and they passed.

@Aggarwal-Raghav
Aggarwal-Raghav force-pushed the HIVE-30127 branch 2 times, most recently from 9823ff3 to 4e10779 Compare October 8, 2026 06:07
@Aggarwal-Raghav
Aggarwal-Raghav marked this pull request as ready for review October 8, 2026 06:09
@Aggarwal-Raghav

Aggarwal-Raghav commented Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

CC @deniskuzZ

deniskuzZ
deniskuzZ reviewed Oct 8, 2026
View reviewed changes
Comment thread iceberg/iceberg-handler/src/test/java/org/apache/iceberg/data/FileHelpers.java Outdated
deniskuzZ
deniskuzZ reviewed Oct 8, 2026
View reviewed changes
Comment thread iceberg/iceberg-handler/src/test/java/org/apache/iceberg/data/FileHelpers.java Outdated
deniskuzZ
deniskuzZ reviewed Oct 8, 2026
View reviewed changes
Comment thread iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/IcebergAcidUtil.java Outdated
@Aggarwal-Raghav
Add apache5-client dependency for Iceberg 1.12.0 S3 support to fix UT
945adfc
@Aggarwal-Raghav
Address review comments:
92847eb 
- Simplify HiveFileWriterFactory constructor by removing unused
  SortOrder and equality delete parameters, and ensure overwrite is
  set on Parquet writers.
- Eliminate deprecated position delete field 2 access across
  IcebergAcidUtil, HiveIcebergCopyOnWriteRecordWriter, and
  HiveIcebergDeleteWriter by using rowDataTemplate directly.
- Sync FileHelpers test utility with upstream Iceberg 1.12.0 reference
  implementation to align delete file format handling.
- Move Parquet file format check inside VariantUtil shredding helper.
@Aggarwal-Raghav
Address UT
8e4b598 
- Add UT TestHMSCatalogAdapterSnapshotMode
- Sync GenericAppenderHelper with upstream Iceberg 1.12.0.
@Aggarwal-Raghav
formattiong
08f9ce4
@Aggarwal-Raghav
Address SonarQube
f9720a9 
- Remove obsolete test testReadAndWriteFormatV2Partitioned_PosDelete_RowSupplied
- Added testConsumeLegacyFormatV2Partitioned_PosDelete_RowSupplied to
  verify Hive can read/consume legacy V2 position deletes containing row
  data via FileHelpers.testOnlyPosDeleteFileWithRow
- Simplify positionDelete helper in TestHiveIcebergCRUD to path and pos
- Remove unused import in TestHMSCatalogAdapterSnapshotMode
@Aggarwal-Raghav

Aggarwal-Raghav commented Oct 12, 2026

Copy link
Copy Markdown
Contributor Author

@Aggarwal-Raghav please review sonar issues, seem related to the current changes

@deniskuzZ , Need 1 suggestion from you on SonaType,

  1. HiveCatalog.java : Its in sync (code wise not formatting wise 😅) with upstream iceberg — NIT , let me know if we want to make this small change.

i would keep the iceberg version in that case, easier later to sync

  1. TestHiveIcebergCRUD.java – testReadAndWriteFormatV2Partitioned_PosDelete_RowSupplied, testReadAndWriteFormatV2Partitioned_PosDelete_RowNotSupplied are using row() and as test name suggest it is testing that functionality. I'll remove these test ?

could you please test if we would be able to consume legacy v2 with row data. do not need tests for write PosDelete_RowSupplied

I've replaced the older UT with a new UT testConsumeLegacyFormatV2Partitioned_PosDelete_RowSupplied , It makes use of FileHelpers.testOnlyPosDeleteFileWithRow (a test class from upstream iceberg, which pre-exist in hive) to generate positional deletes with rows

deniskuzZ
deniskuzZ reviewed Oct 12, 2026
View reviewed changes
Comment thread iceberg/iceberg-handler/src/test/java/org/apache/iceberg/data/FileHelpers.java
}

return writer.toDeleteFile();
public static Pair<DeleteFile, CharSequenceSet> writeDeleteFile(

@deniskuzZ deniskuzZ Oct 12, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need this helper with hardcoded V2?

@Aggarwal-Raghav Aggarwal-Raghav Oct 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, so, FileHelpers and GenericAppenderHelper are sync fully from upstream #6856 (comment)

Attaching URL for your reference:

  1. GenericAppenderHelper: https://github.com/apache/iceberg/blob/apache-iceberg-1.12.0/data/src/test/java/org/apache/iceberg/data/GenericAppenderHelper.java
  2. https://github.com/apache/iceberg/blob/apache-iceberg-1.12.0/data/src/test/java/org/apache/iceberg/data/FileHelpers.java

I'll try to removed them if possible and replace them with iceberg-data test scope dependecy in a follow-up PR

@deniskuzZ deniskuzZ Oct 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, meanwhile if some introduced methods are dead- can we kill them?

@Aggarwal-Raghav Aggarwal-Raghav Oct 12, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I get your point but lets keep them.
I'm planning to raise other iceberg 1.12.0 PR's for new features introduced in iceberg. By that time, we feel that these methods can be deleted, will raise a follow-up PR.

@deniskuzZ deniskuzZ Oct 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, i'll merge once we get a green build. Thanks again for addressing the review comments!

deniskuzZ
deniskuzZ reviewed Oct 12, 2026
View reviewed changes
Comment thread iceberg/iceberg-handler/src/test/java/org/apache/iceberg/data/GenericAppenderHelper.java
File file = tmp.newFile();
Assert.assertTrue(file.delete());
File file =
temp.resolve("generic-appender-partition-test-" + UUID.randomUUID()).toFile();

@deniskuzZ deniskuzZ Oct 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you please move to constant "generic-appender-partition-test-"

deniskuzZ
deniskuzZ reviewed Oct 12, 2026
View reviewed changes
Comment thread iceberg/iceberg-handler/src/test/java/org/apache/iceberg/data/GenericAppenderHelper.java
GenericFileWriterFactory.Builder builder =
new GenericFileWriterFactory.Builder(table).dataFileFormat(format);

// Push down ORC related settings to appender if there are any

@deniskuzZ deniskuzZ Oct 12, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Push down ORC and Parquet ?

@sonarqubecloud

sonarqubecloud Bot commented Oct 12, 2026

Copy link
Copy Markdown

Quality Gate Passed Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
1.3% Duplication on New Code

See analysis details on SonarQube Cloud

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@deniskuzZ deniskuzZ deniskuzZ approved these changes

Assignees

No one assigned

Labels

tests pending

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@Aggarwal-Raghav @deniskuzZ @asf-ci-hive