fix(trino): read uncompacted MDT HFILE log deltas and guard index pruning#19298
Draft
voonhous wants to merge 7 commits into
Draft
fix(trino): read uncompacted MDT HFILE log deltas and guard index pruning#19298voonhous wants to merge 7 commits into
voonhous wants to merge 7 commits into
Conversation
…-105)
Squash of the 55-commit review history. The connector implementation moves
into the Hudi repo as the hudi-trino module (published as
org.apache.hudi:hudi-trino), targeting Trino SPI 481 on JDK 25, behind the
opt-in hudi-trino Maven profile. The Trino repo keeps only a thin trino-hudi
shim depending on the published artifact. Removes hudi-trino-bundle and the
trino{base,coordinator,worker} docker modules.
Co-authored-by: vamsikarnika <vamsikarnika@gmail.com>
Co-authored-by: Aditya Goenka <ad1happy2go@gmail.com>
… parquet reader MOR tables written at table version 10 store delta-log blocks as native parquet files (RFC-103). Reading one calls HoodieIOFactory.getFileFormatUtils(PARQUET).readFooter(...) to pull the log-block header out of the parquet footer, but HudiTrinoIOFactory stubbed getFileFormatUtils out, so any realtime read of a v10 MOR table failed with "FileFormatUtils not supported in HudiTrinoIOFactory" (surfaced by TestHudiCustomMerger and TestHudiCustomMergerEndToEnd, which write their tables at the default version). Add HudiTrinoParquetFileFormatUtils, implementing readFooter with Trino's own parquet reader (MetadataReader.readFooter -> FileMetadata.getKeyValueMetaData), mirroring ParquetUtils.readFooter semantics, and return it from HudiTrinoIOFactory.getFileFormatUtils for PARQUET; non-parquet native log formats now throw a clear message. Record decoding already flows through Trino's page-source reader, so the footer read was the only missing piece. No new dependency is added (hudi-hadoop-common stays excluded).
The reader context ignored its path/schema arguments and re-served the pre-built base page source, so RFC-103 native (parquet) log records were never read and v10 MOR reads returned base-only values. Build a Trino parquet page source on demand for native log files, projected on the file-group reader's requiredSchema with predicate pushdown off, via a factory injected from HudiPageSourceProvider. Synthesize VARCHAR handles for Hudi meta columns the file-group reader adds to requiredSchema beyond the connector projection (e.g. _hoodie_commit_time); all meta columns are UTF8 strings on disk and parquet columns are resolved by name, so the placeholder ordinal is never read (refactor tracked by apache#19249). Custom mergers must be projection compatible: a non-projection-compatible merger makes the file-group reader demand the full table schema, whose unprojected data columns cannot be typed at this layer; fail such reads with a clear NOT_SUPPORTED error and declare isProjectionCompatible() on the test mergers, which merge on their declared mandatory fields only. Also adapt the TestHudiSmokeTest footer-read call site to the parameterized createPageSource helper.
voonhous
changed the title
voonhous
marked this pull request as draft
July 15, 2026 09:03
Collaborator
voonhous
force-pushed
the
19279--mdt-hfile-log-deltas
branch
from
July 15, 2026 09:12
545ca14 to
6d3bb86
Compare
3 tasks
…K 25 compile Master commit b0a2563 declares Lombok in the root pom's annotationProcessorPaths (a parallel-build deflake). hudi-trino inherits that path but compiles on JDK 25, where Lombok 1.18.36 crashes javac with ExceptionInInitializerError: com.sun.tools.javac.code.TypeTag :: UNKNOWN -- this broke the hudi-trino CI on every PR merge ref once that commit landed. hudi-trino uses no annotation processors, so set proc=none and override the inherited processor path away.
Fixes apache#19249, including the scope update in issue comment 4937143911; relates to the base read path of apache#18837. The file-group reader can require merge columns the query projection does not carry: Hudi meta fields like _hoodie_commit_time, delete markers, record-key data columns, a custom merger's mandatory fields, or the whole table schema when a CUSTOM merger is not projection compatible. buildRequiredColumnHandles previously hand-synthesized a HIVE_STRING/VARCHAR handle behind a HOODIE_META_COLUMNS guard and threw NOT_SUPPORTED for anything else, and the base-file read silently returned projection-only records that made such mergers see nulls. Log side: - Every required-schema field carries its real Avro type from the table schema, so unresolved handles are typed directly from the field (NativeLogicalTypesAvroTypeBlockHandler + HiveTypeTranslator) via the new HudiUtil.toColumnHandle; the synthesis, guard and exception are gone. trino-hive-formats moves from runtime to compile scope. Base side: - HudiPageSourceProvider mirrors the file-group reader's decision: HudiUtil.resolveMergeModeAndStrategyId applies the same version-gated inference as hudi-common (merge mode below table version 9, strategy id below version 8), and when the resolved CUSTOM merger is not projection compatible (interface default; also every payload-based table, since the payload strategy resolves HoodieAvroRecordMerger), the read projection expands to every table-schema column via HudiUtil.appendMissingSchemaColumns. Matches Spark; the extra I/O is inherent to such mergers. - getMergeRequiredColumnHandles predicts every column FileGroupReaderSchemaHandler.getMandatoryFieldsForMerging can demand: ordering columns under the inferred (not raw) merge mode, _hoodie_is_deleted and _hoodie_operation unconditionally (dropped when not table columns), the custom delete key when both hoodie.payload.delete.field and .marker are set, and the record-key data columns when hoodie.populate.meta.fields is false. - The base read reuses the pre-built page source, so its projection MUST cover requiredSchema: HudiTrinoReaderContext now guards that invariant and fails with HUDI_SCHEMA_ERROR naming any missing columns instead of merging with nulls. MOR splits with log files but no base file fail with a clear NOT_SUPPORTED instead of a bare NoSuchElementException. Cleanups in the same area: - Replace SynthesizedColumnHandler with PrefilledColumnValues, a thin per-split adapter over trino-hive's HiveUtil.getPrefilledColumnValue. Fixes $partition rendering multi-key partitions in hash order, handles the hive null marker (backslash-N) in partition values, adds previously unsupported partition-key types, and deletes SynthesizedColumnStrategy plus the hand-rolled appendPartitionKey dispatch. - Remove dead code (HudiTrinoReaderContext.colToPosMap and dataHandles, HudiPageSource.readerContext); make fields private final. - remapColumnIndicesToPhysical throws a diagnosable TrinoException instead of an autoboxing NPE for columns absent from the parquet file schema, and lowercases with Locale.ROOT. Tests: - TestHudiNonProjectionCompatibleMerger + NonProjectionCompatibleRankMerger (no isProjectionCompatible/getMandatoryFieldsForMerging overrides) + initializer: narrow-projection MOR queries that never project merge_rank, laid out so one key's winning rank is on the log record and the other's on the base record; sum(value)=199 discriminates the correct merge from base-only (110) and newest-wins (103). - Unit coverage: toColumnHandle, appendMissingSchemaColumns, usesNonProjectionCompatibleMerger, resolveMergeModeAndStrategyId (table versions 6/8/9), merge-required column prediction, PrefilledColumnValues (incl. the $partition ordering fix), and the typed missing-column remap error (both remap tests). - Existing projection-compatible merger tests stay as fast-path regression coverage; stale MaxRankRecordMerger javadoc refreshed.
…nd commit-time ordering Fixes apache#18898 (follow-up from the RFC-105 review of HudiTrinoReaderContext.getRecordMerger): the merge-mode dispatch had no end-to-end coverage where the merger choice matters. Adds runtime-written v10 MOR tables and snapshot-read tests for: - Deletes under EVENT_TIME_ORDERING (mor_deletes): hard deletes -- which at v10 are native delete log files read back through the connector's own getFileRecordIterator with the synthetic delete-log schema, a previously untested path enabled by the required-column resolution this branch stacks on -- plus _hoodie_is_deleted soft deletes, an OBSOLETE soft delete and an OBSOLETE update (lower ordering value) that must LOSE against the base row. - COMMIT_TIME_ORDERING (mor_commit_time): a lower-ordering update that must WIN (latest write), the exact mirror of the event-time case, discriminating OverwriteWithLatestMerger from event-time merging. - Payload-driven semantics without any hudi.record-merger-impls property: AWSDmsAvroPayload (v10-translated to COMMIT_TIME + delete-key props; Op='D' log records delete rows at merge time), OverwriteNonDefaultsWithLatestAvroPayload (v10-translated to PARTIAL_UPDATE_MODE=IGNORE_DEFAULTS; null update columns keep stored values), and a user-defined SummingTestPayload riding the payload-based CUSTOM strategy whose merged value (old + new) proves the payload's combineAndGetUpdateValue executed at read time. Test rows are written with the pass-through HoodieAvroPayload (not a BaseAvroPayload) so delete-flagged rows land as data records and every merge decision happens at read time from the table config. Bugs found by these tests and fixed here: - HudiUtil.mergeRequiredColumnNames read the custom delete key/marker from raw table props, but v9+ table creation persists them PREFIXED (hoodie.record.merge.property.*). The file-group reader strips the prefix via getTableMergeProperties before DeleteContext reads the plain keys, so its required schema included the delete column while the connector's base-read prediction missed it and the base-read projection guard failed every narrow projection on such tables. Now resolved the same way the reader does: table merge properties first, raw props as fallback. Unit-tested in TestHudiMergeRequiredColumns. - buildColumnHandles matched predicted merge columns against metastore data columns case-sensitively; the prediction carries the table schema's casing (AWSDms hardcodes 'Op') while metastore names are lowercased, so the handle was never built. Matching is now case-insensitive. - buildRequiredColumnHandles emitted projection handles with the lowercased metastore name on the log projection, but hudi-common reads merge fields off log records by the table schema's exact field name (Avro lookups are case-sensitive), so e.g. the DMS delete marker was never seen on log records. Handles are now re-labeled with the required-schema field's exact name; parquet columns resolve case-insensitively either way. - count(*) over a MoR split with log files failed with "Failed to construct Avro schema": the query projects no columns, and HudiPageSource's serializer fed the empty projection to AvroHiveFileUtils.determineSchemaOrThrowException, which rejects an empty column list. HudiUtil.constructSchema now short-circuits an empty projection to an empty record schema; the page source already counts positions fine with zero channels. HudiUtilTest's empty-columns test now pins the empty-record schema instead of the former exception. - Payload-based merging read garbage on the summing table ([, -44] instead of [k1, 109]): records handed to hudi-common carried a schema reconstructed from Hive metastore types (every column a nullable union, fields in projection order), while the file-group reader tracks its required schema for those records. BaseAvroPayload round-trips the newer record through Avro binary with the tracked schema, so the structural mismatch misaligned the binary decode. HudiTrinoReaderContext now builds the log-side AND base-side serializers with requiredSchema itself, via a new HudiAvroSerializer constructor that takes the record schema explicitly and maps page channels to record fields BY NAME (the base projection's order can differ from the required schema's field order; containment holds in both directions through the existing base-read guard and generateRequiredSchema always containing the requested schema). This also aligns record layouts with the positions hudi-common derives from the tracked schema (partial-update merging, _hoodie_operation lookup). - (hudi-common, engine-agnostic) The file-group reader passed its raw input props to initRecordMerger, the MERGE_TYPE lookup and the schema handler instead of the ConfigUtils.getMergeProps view, which layers the table's prefixed merge properties (hoodie.record.merge.property.*) in as plain keys -- a regression from commit 19961e7 replacing the previous in-place putAll with a copy. DeleteContext therefore missed the translated DMS delete key/marker, the delete column was not a mandatory merge field, and Op='D' log records merged as regular data updates instead of deletes on snapshot reads. Fixed in HoodieFileGroupReader and HoodieLsmFileGroupReader alike by passing this.props downstream. Also re-ports a shim-lineage guard (trino repo commit 78209f83b50) that the RFC-105 migration squash predates: HudiAvroSerializer's default constructor now maps channels past hidden (synthesized, split-prefilled) columns instead of relying on serialize() never being called with them; with identity positions a hidden column would shift every later value into the wrong field and overrun the record. Also replaces the TODO(apache#18898) in getRecordMerger with a pointer to the new suites.
…ning Fixes apache#19279: with the connector default hudi.metadata-enabled=true, any query on a table whose metadata table (MDT) has uncompacted delta commits failed with 'Native HFILE log files are not supported by the Hudi Trino connector'. Since MDT compaction fires only every hoodie.metadata.compact.max.delta.commits (default 10) deltas, actively written tables spend most of their life in that state; the E2E testcontainers pipeline had to set hudi.metadata-enabled=false to pass. - HudiTrinoIOFactory.getFileFormatUtils now returns hudi-common's HFileUtils for HFILE. The native-log read path (HoodieNativeLogFileReader) needs only its readFooter, and HFileUtils is hadoop-free (it decodes via hudi-io's native HFile reader), so it is safe in the Trino runtime -- unlike parquet, whose ParquetUtils lives in the hadoop-excluded hudi-hadoop-common and needs the Trino-native HudiTrinoParquetFileFormatUtils. The reader-factory side (newHFileFileReader) was already implemented; it is how compacted MDT base HFiles are read today. MDT reads never touch HudiTrinoReaderContext. - Guard the partition-stats pruning call in HudiBackgroundSplitLoader.getPartitionInfos: any pruning failure now logs and proceeds unpruned, mirroring the metastore fallback the MDT partition listing directly above it already has. Index pruning is an optimization and must never fail the query. The per-file-slice indexes (column-stats, record-level, secondary) already swallow failures via their future+catch pattern. - Guard the MDT-backed file-system-view load in HudiSnapshotDirectoryLister: on failure, close the partial view and fall back to a direct-listing view built over an MDT-disabled table metadata. This was the second unguarded synchronous MDT read. Tests: UncompactedMetadataHudiTablesInitializer writes a partitioned COW table with an ENABLED metadata table (column-stats + partition-stats indexes) and compact.max.delta.commits=100 -- the inverse of the zip fixtures' compact.max.delta.commits=1, whose always-compacted MDTs are why nothing caught this -- leaving native HFILE log deltas in the files/column_stats/ partition_stats MDT partitions. MDT HFILE writing is native (hudi-io HFileWriterImpl); the 'requires hbase' note in older initializers is stale. TestHudiUncompactedMetadataTable covers MDT-backed listing, MDT-on vs MDT-off result equality, partition-stats pruning (the exact crash path) and column-stats file skipping over the uncompacted stats. The E2E follow-up (separate branch trino-e2e-testcontainers): drop the hudi.metadata-enabled=false workaround from the Trino catalog so the pipeline runs MDT-on.
voonhous
force-pushed
the
19279--mdt-hfile-log-deltas
branch
from
July 15, 2026 16:49
6d3bb86 to
b1352ed
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #19295 (-> #19288 -> #18837) -- only the head commit is new here.
Describe the issue this Pull Request addresses
Closes #19279
With the connector default
hudi.metadata-enabled=true, any query on a table whose metadata table (MDT) has uncompacted delta commits fails with "Native HFILE log files are not supported by the Hudi Trino connector". MDT compaction only runs every 10 delta commits by default, so an actively written table spends most of its life in that state -- default Spark writer plus default catalog config meant every read failed. The E2E pipeline currently setshudi.metadata-enabled=falseto work around it.Summary and Changelog
The gap turned out to be small: the connector already reads HFile base files (compacted MDT) through hudi-io's native reader, and the native-log read path only needs
getFileFormatUtils(HFILE).readFooter. hudi-common'sHFileUtilsis hadoop-free (it decodes via hudi-io), so unlike parquet there is no need for a Trino-native implementation.HudiTrinoIOFactory.getFileFormatUtilsnow returnsHFileUtilsfor HFILE, making uncompacted MDT deltas (native HFILE log files) readable.HudiBackgroundSplitLoaderno longer fails the query: on error it logs and proceeds unpruned, same as the metastore fallback the partition listing right above it already had. This was the unguarded path in the issue's stack.HudiSnapshotDirectoryLister(the other unguarded synchronous MDT read) falls back to direct file listing on error.Tests: a new initializer writes a partitioned COW table with the MDT enabled and
compact.max.delta.commits=100-- the inverse of the zip fixtures'=1, which is why nothing caught this -- leaving real HFILE log deltas in thefiles/column_stats/partition_statsMDT partitions.TestHudiUncompactedMetadataTablecovers MDT-backed listing, MDT on/off result equality, partition-stats pruning (the exact crash) and col-stats file skipping. Side note: MDT writing needs no hbase anymore (hudi-io'sHFileWriterImpl), so the "requires hbase" comment in the older initializers is stale.Impact
Tables with uncompacted MDTs are readable under default configs, and index pruning / MDT listing degrade gracefully instead of failing queries. No API or config changes. Follow-up on the E2E branch: drop the
hudi.metadata-enabled=falsecatalog workaround so the pipeline runs MDT-on.Risk Level
low -- the format support is a one-line dispatch to existing hudi-common/hudi-io code, and the two guards mirror fallback patterns already in the connector; all pinned by the new tests.
Documentation Update
none
Contributor's checklist