Repository navigation
[core] Do not push filters on row tracking fields into append file stats and readers - #10959
Conversation
JingsongLi
left a comment
JingsongLi
left a comment
There was a problem hiding this comment.
Requirement fit: SUPPORTED. Implementation: FINDINGS.
Filtering row-tracking metadata on ordinary append tables is a real supported read path. The normal JDK 8 core reactor package passes all 14 targeted cases (RowTrackingTableTest, RawFileSplitReadTest and RowTrackingCopiedFilesTest); upstream CI is green. A real-table filter-plus-limit case still returns incorrect results as described below. This is an incomplete fix of the stated filtering requirement, rather than a claim that this query already worked on the baseline.
Verified fixed at a5805de. Requirement fit: SUPPORTED; implementation: CLEAN on this reviewed head. All 14 targeted JDK 8 core tests pass with normal quality checks. The updated row-tracking test covers _ROW_ID and _SEQUENCE_NUMBER with and without LIMIT 1; the original incorrect-empty-result reproduction now passes. The deferred-filter flag prevents early format/file-index TopN and limits and the final reader cap. The original P2 is resolved, so I am withdrawing this blocking review. The new head's CI is still running.
JingsongLi
left a comment
JingsongLi
left a comment
There was a problem hiding this comment.
Requirement fit: SUPPORTED. Implementation: FINDINGS.
The equality-filter + limit case is fixed in this revision. One remaining P2 concerns predicates with both user-column and row-tracking inputs, as detailed inline.
Validation: all 14 targeted tests (RowTrackingTableTest, RawFileSplitReadTest, and RowTrackingCopiedFilesTest) pass in the normal JDK 8 core reactor package build. An additional table-API regression test reproduces the mixed-input + schema change + limit failure and passes when the exclusion uses field-set intersection. Full Spark/Flink suites were not run locally.
cffedb3 to
9f9423c
Compare
For a row tracking append table without data evolution, _ROW_ID and _SEQUENCE_NUMBER come from the file meta, not from the file stats or the file columns. A filter on them, as from the row_tracking system table, failed scan planning on the stats, or made the file reader prune every row. Leave any filter that references these fields, alone or mixed with data columns, out of the stats filter and the format pushdown, so it only runs on the rows. When such a filter is left out, the reader also skips its early TopN and limit, so they cannot drop the matching rows. Fixes apache#9264.
9f9423c to
0f0eb63
Compare
Purpose
Fixes #9264.
On an append table with
row-tracking.enabledbut no data evolution, a filter on_ROW_IDor_SEQUENCE_NUMBER(for exampleSELECT * FROM t$row_tracking WHERE _ROW_ID = 0) breaks the read. Scan planning fails withAssertionError: index (2) should < 2, orFind no field _ROW_IDafter a schema change. When planning gets through, the file reader prunes every row and returns nothing.These values come from file metadata, not from file stats or columns. Such filters now stay out of stats and format pushdown and are evaluated on rows, as data evolution tables already do.
Tests