HIVE-29798: NumberFormatException while reading a table having UNION subdirs after ACID conversion

[abstractdog]

What changes were proposed in this pull request?

Fix a NumberFormatException in the ACID reader when a table is converted to full ACID after being loaded via INSERT ... UNION ALL.

Concretely, VectorizedOrcAcidRowBatchReader walks from the split's path up to the table/partition root looking for a base_* / delta_* / delete_delta_* ancestor. In the original code the "else" branch — any non-base_* parent — was unconditionally fed to AcidUtils.ParsedDeltaLight.parse(parent), which substrings past the delta_ prefix and calls Long.parseLong. When the parent is a UNION-ALL leftover such as HIVE_UNION_SUBDIR_15/, that produces:

java.lang.NumberFormatException: For input string: "NION"

The fix guards the parse with startsWith(DELTA_PREFIX) || startsWith(DELETE_DELTA_PREFIX) and simply keeps walking up on any other name. This mirrors the same guard already present in the sibling non-vectorized reader OrcRawRecordMerger.TransactionMetaData#findWriteIDForSynthetcRowIDs (OrcRawRecordMerger.java:1268-1292), which explicitly comments that HIVE_UNION_SUBDIR_<N> is a meaningful path for nonAcid→acid tables.

Files touched (main change):

• ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java — add the isBase / isDelta guard around ParsedDeltaLight.parse.

New test:

• ql/src/test/org/apache/hadoop/hive/ql/metadata/TestUnionAllToAcidConversion.java — 8 tests covering {unpartitioned, partitioned} × {CONVERT TO ACID, SET TBLPROPERTIES('transactional'='true')} × {write-time flatten off, write-time flatten on}.

Why are the changes needed?

Because reading a table after converting it to ACID currently crashes if the pre-conversion data was written by a UNION-ALL. Reproducer (in the added test file):

CREATE TABLE t (a int, b int) STORED AS ORC TBLPROPERTIES ('transactional'='false');
-- Writes go under warehouse/t/HIVE_UNION_SUBDIR_1..3/000000_0
INSERT INTO t
  SELECT k, sum(v) FROM src WHERE k=1 GROUP BY k UNION ALL
  SELECT k, sum(v) FROM src WHERE k=2 GROUP BY k UNION ALL
  SELECT k, sum(v) FROM src WHERE k=3 GROUP BY k;

ALTER TABLE t CONVERT TO ACID;           -- OR: ALTER TABLE t SET TBLPROPERTIES ('transactional'='true');
SELECT COUNT(*) FROM t;                  -- boom: NumberFormatException: "NION"

Both ACID DDL paths (CONVERT TO ACID and SET TBLPROPERTIES('transactional'='true')) are documented as metadata-only flips — the pre-conversion on-disk layout is preserved. TestTxnNoBuckets.testToAcidConversionMultiBucket explicitly relies on that: it asserts the HIVE_UNION_SUBDIR_<N>/ layout survives the conversion and verifies specific per-subdir ROW__ID assignments. So the reader is the correct place to handle the pre-conversion layout — trying to rewrite files at conversion time would break that established contract (and, when I tried it, actually broke TestTxnNoBuckets).

The bug is also reachable through the "old" UpgradeTool-generated conversion scripts, which emit ALTER TABLE ... SET TBLPROPERTIES ('transactional'='true') (see ql/src/java/org/apache/hadoop/hive/ql/util/UpgradeTool.java:524,574).

Does this PR introduce any user-facing change?

Yes — a bug fix. Users who convert a non-ACID table to full ACID via either ALTER TABLE ... CONVERT TO ACID or ALTER TABLE ... SET TBLPROPERTIES ('transactional'='true') after having loaded it with INSERT ... UNION ALL can now read the resulting table without hitting NumberFormatException: For input string: "NION". No API, DDL syntax, or on-disk layout changes.

How was this patch tested?

New JUnit 5 test class ql/src/test/org/apache/hadoop/hive/ql/metadata/TestUnionAllToAcidConversion — 8 tests, all pass with the fix, cover the full matrix:

• write-time flatten OFF (hive.tez.union.flatten.subdirectories=false) — the case this PR fixes:

  • testUnionAllInsertThenConvertToAcid — unpartitioned, CONVERT TO ACID.
  • testUnionAllInsertThenSetTblpropertiesAcid — unpartitioned, SET TBLPROPERTIES.
  • testPartitionedUnionAllInsertThenConvertToAcid — partitioned, CONVERT TO ACID.
  • testPartitionedUnionAllInsertThenSetTblpropertiesAcid — partitioned, SET TBLPROPERTIES.

  Each asserts the pre-conversion layout literal (e.g. /t/HIVE_UNION_SUBDIR_{1,2,3}/000000_0), asserts the same layout post-conversion (metadata-only flip), and finally asserts SELECT COUNT(*) returns 3 — which is what exercises the reader-side fix end-to-end.

• write-time flatten ON (hive.tez.union.flatten.subdirectories=true) — orthogonal but nearby behavior; four tests pin the current state:

  • testUnionAllInsertWithFlattenThenConvertToAcid
  • testUnionAllInsertWithFlattenThenSetTblpropertiesAcid
  • testPartitionedUnionAllInsertWithFlattenThenConvertToAcid
  • testPartitionedUnionAllInsertWithFlattenThenSetTblpropertiesAcid

  These document a second bug that's out of scope here: MoveTask.flattenUnionSubdirectories produces <index>_000000_0 files (three numeric parts) which don't match the metastore's ORIGINAL_PATTERN ([0-9]+_[0-9]+), so a subsequent ACID conversion is rejected by TransactionalValidationListener.validateTableStructureForPath with "Unexpected data file name format". The tests assert exactly that failure and include a comment pointing at what should change to make them assert successful conversion instead.

Regression check — the two tests most likely to break:

TestTxnNoBuckets#testToAcidConversionMultiBucket           ✓  (pins the "layout preserved through conversion" contract)
TestVectorizedOrcAcidRowBatchReader                        ✓  (17/17 — the class being changed)
TestFileSinkOperator                                       ✓  (9/9   — flatten call sites)
TestUnionAllToAcidConversion                               ✓  (8/8)

[sonarqubecloud]

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud