Hive: document hive2-compatible and fix commit locks on Hive 2.1 - #3927
Hive: document hive2-compatible and fix commit locks on Hive 2.1#3927kevinjqliu wants to merge 3 commits into
Conversation
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Hive 2.1.0 rejects a lock component whose operation type arrives as the default UNSET, and the generated Thrift client always sends that default. The commit lock is not part of a metastore transaction, so send NO_TXN, which every Hive release accepts. Verified against Hive 2.0.0, 2.1.0, 2.3.2, 3.1.3, 4.0.0, 4.0.1, 4.1.0 and 4.2.1 metastores. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
this addresses my comment about testing hive 2 #3924 (comment) Claude helped test all the different versions of hive and it all passed. |
There was a problem hiding this comment.
🟡 Changes recommended
There are review comments to address around clarifying an internally inconsistent code comment and adding/considering coverage to prevent regressions of the Hive 2.1.0 lock fix.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR is a follow-up to the Hive 4.2.1 CI update, improving Hive metastore compatibility by clarifying configuration guidance for Hive 2.x and fixing a Hive 2.1.0 lock-component incompatibility that can break commits.
Changes:
- Document that
hive.hive2-compatible: trueshould be enabled when using a Hive 2.x metastore (Hive 3+ works by default). - Set the Hive commit lock component’s
operationTypetoDataOperationType.NO_TXNto avoid Hive 2.1.0 rejecting the defaultUNSET.
File summaries
| File | Description |
|---|---|
| pyiceberg/catalog/hive.py | Explicitly sets operationType=NO_TXN on the metastore lock component to restore commit compatibility with Hive 2.1.0. |
| mkdocs/docs/configuration.md | Updates Hive catalog configuration docs to more clearly instruct Hive 2.x users to enable hive.hive2-compatible. |
Review details
- Files reviewed: 2/2 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| # Iceberg commits do not open a metastore transaction, so the lock is marked NO_TXN. Setting it explicitly | ||
| # also matters for Hive 2.1, which rejects a lock component left at the default UNSET operation type. |
| type=LockType.EXCLUSIVE, | ||
| dbname=database_name, | ||
| tablename=table_name, | ||
| operationType=DataOperationType.NO_TXN, |
Follow up to #3924.
Rationale for this change
Two small things after moving CI to Hive 4.2.1:
hive.hive2-compatiblenow also selects the legacy thrift calls, so document it as "set this if you're on Hive 2". Hive 3+ works by default.operationType=NO_TXNon the commit lock. Hive 2.1.0 rejects a lock component left at the defaultUNSET(HMS bug, relaxed in 2.1.1), which broke every commit on that version. Every other version acceptsNO_TXN.Related: #1222, #1653
Are these changes tested?
Ran the catalog against a metastore for each Hive line: create, load, list, append, rename, drop.
The wrong setting fails right away with
Invalid method name, so a misconfig is obvious. Integration tests pass against 4.2.1.Are there any user-facing changes?
Yes. Since #3924, Hive 2.0-2.2 users need
hive.hive2-compatible: true(2.3 has both sets of calls but still needs the flag for timestamptz columns). Commits now work on Hive 2.1.0.🤖 Generated with Claude Code