feat: Native Delta Lake scan via delta-kernel-rs - #3932

Closed

schenksj wants to merge 80 commits into

apache:mainfrom

schenksj:delta-kernel-phase-1

Commits on Apr 13, 2026

feat: add native Delta Lake scan via delta-kernel-rs

schenksj
and
claude
committed
10dd91a
test: Delta native scan test suites

schenksj
and
claude
committed
6980fa1
bench: Delta benchmarks and TPC runner infrastructure

schenksj
and
claude
committed
0a21380
ci/docs: Delta CI workflow and documentation

schenksj
and
claude
committed
2cef606
feat: expand kernel predicate pushdown with IN and Cast support

schenksj
and
claude
committed
c0ada8d
docs: add IN/NOT IN and Cast to supported predicates list

schenksj
and
claude
committed
2b8dedd
fix: address CI linting and security feedback

schenksj
and
claude
committed
64e1f3e

Commits on Apr 14, 2026

fix: use Hadoop Path when parsing input file URIs in native Delta scan

schenksj
and
claude
committed
c97b60e
test: add Delta Lake regression suite mirroring the Iceberg pattern

schenksj
and
claude
committed
3e4b6a0
fix: distinguish CometDeltaNativeScan instances across snapshot versions

schenksj
and
claude
committed
29361d6
fix: use correct Delta artifact ID for Spark 3.4 test dependency

schenksj
and
claude
committed
ee5a375
test(delta-regression): robustness fixes + DELTA_JAVA_HOME support

schenksj
and
claude
committed
bf38729
test(delta-regression): make Delta 2.4.0 diff install Comet extensions

schenksj
and
claude
committed
db4b6eb
Merge remote-tracking branch 'upstream/main' into delta-kernel-phase-1
schenksj
committed
d8aa2bb
fix: populate InputFileBlockHolder for Delta native scans so Delta MERGE/UPDATE/DELETE work

schenksj
and
claude
committed
75e404d
test(delta-regression): make spark/test actually run on modern JDKs

schenksj
and
claude
committed
0dbb807

Commits on Apr 15, 2026

feat(delta): close major native-scan coverage gaps + row tracking

schenksj
and
claude
committed
297d857
chore(delta-regression): drop row-id-lookup targeted test

schenksj
and
claude
committed
340a594

Commits on Apr 16, 2026

fix(delta-regression): force UTC JVM timezone; use rootPaths for scheme check

schenksj
and
claude
committed
24427b5
docs(delta): refresh support matrix; clarify cloud-fetch guard

schenksj
and
claude
committed
ff352e4
fix(delta): translate column-mapping names on the pre-materialised-index path

schenksj
and
claude
committed
9e88ac0
fix(delta): code review hardening + CI workflow registration

schenksj
and
claude
committed
6b276b1

Commits on Apr 17, 2026

fix(delta): nested column mapping, checkpoint errors, TZ partition values

schenksj
and
claude
committed
7844db1
fix(delta): MERGE + column mapping name mode unblocks field metadata

schenksj
and
claude
committed
607e89b
fix(delta): session-TZ timestamp partition values + z-order shuffle bytes

schenksj
and
claude
committed
a4dfc3a
fix(delta): preserve DV filter wrapper when scan would fall back

schenksj
and
claude
committed
a01949a
fix(delta): alias output_rows to numOutputRows for streaming progress

schenksj
and
claude
committed
08f1ed8
refactor(delta): code review hardening round 2

schenksj
and
claude
committed
92f1da0
test(delta): add CometDeltaRoundTripSuite for round-2 fix coverage

schenksj
and
claude
committed
2803322
test(delta): add dev/run-delta-all-failures.sh

schenksj
and
claude
committed
bbc57dc
fix(delta): broaden TIMESTAMP partition-value parsing + skip nested-access pushdown

schenksj
and
claude
committed
98aa8b2

Commits on Apr 18, 2026

fix(delta): preserve DV wrapper for PreparedDeltaFileIndex too

schenksj
and
claude
committed
e65f6ed
fix(delta): extend DV fallback to any scan with is_row_deleted + PreparedDeltaFileIndex DVs

schenksj
and
claude
committed
71313d2
test(comet): plain-Parquet repro for Utf8<=Int32 bug establishes the bug is NOT Comet-core

schenksj
and
claude
committed
6804916
fix(delta): column-mapping scan hardening (fixes drop-column-with-constraints)

schenksj
and
claude
committed
f917f67
fix(delta): DV-aware UPDATE bookkeeping + name-mode partition keys + shallow-clone mock URLs

schenksj
and
claude
committed
609b904

Commits on Apr 19, 2026

fix(delta-regression): unwrap CometScanExec in OptimizeGeneratedColumnSuite's getPushedPartitionFilters

schenksj
and
claude
committed
2508588

Commits on May 1, 2026

fix(delta-regression): stop overriding dvFileNamePrefix in Delta test harness

schenksj
and
claude
committed
da19481
feat(delta): file-splitting via byte-range tasks in CometDeltaNativeScan

schenksj
and
claude
committed
bb35abf
fix(delta): decline scan when Delta synthetic columns are in output

schenksj
and
claude
committed
b1fce4f
fix(aqe): preserve a logical link on Comet exchange wrappers

schenksj
and
claude
committed
7d8c3b0

Commits on May 2, 2026

fix(delta-regression): replace `spark%dir%prefix` temp-dir prefix in test harness

schenksj
and
claude
committed
a0a5caa
fix(delta-regression): drop both path-prefix test shims

schenksj
and
claude
committed
24fb5e4
fix(delta): decline native scan when ignoreMissingFiles is enabled

schenksj
and
claude
committed
9f9335e
feat(parquet): lazy ignoreMissingFiles in the native Delta scan

schenksj
and
claude
committed
8b6fdaf
chore(dev): add FAST=1 mode to run-delta-regression.sh

schenksj
and
claude
committed
ff5c92f
fix(aqe): preserve a logical link on every Comet exec, not just exchanges

schenksj
and
claude
committed
7d52de7
fix(delta): single-URI-encoded table roots and PreparedDeltaFileIndex routing

schenksj
and
claude
committed
2b09698
fix(delta): always send fully-encoded URI to native scan

schenksj
and
claude
committed
f8771bc
chore: gitignore .claude/ scheduled-tasks lock

schenksj
and
claude
committed
2b0d747
fix(delta): gate physicalName synthesis on table column-mapping mode

schenksj
and
claude
committed
bf59576
fix(delta): don't override user-set useMetadataRowIndex

schenksj
and
claude
committed
ca26110
fix(delta): decline native Delta scan when input_file_name is referenced

schenksj
and
claude
committed
d203466

Commits on May 3, 2026

fix(delta): bin-pack scan tasks into Spark partitions

schenksj
and
claude
committed
845c1e2
feat(delta): drop typeWidening fallback gate

schenksj
and
claude
committed
0d67db2
feat(delta): drop rowTracking fallback gate

schenksj
and
claude
committed
1c12945
feat(delta): wire parquet encryption through native Delta scan

schenksj
and
claude
committed
9c1e64e
chore(delta): remove dead PreparedDeltaFileIndex DV check

schenksj
and
claude
committed
3f8f227
fix(delta): use preparedScan.files for PreparedDeltaFileIndex

schenksj
and
claude
committed
197ee4a
fix(delta): parse TIMESTAMP_NTZ partition values without session TZ shift

schenksj
and
claude
committed
5c00233
fix(delta): expose numFiles metric alias + patch DeltaSuite scan-collect

schenksj
and
claude
committed
1dbfc8b
fix(delta): patch merge-metrics shim to accept Comet's 2-file output

schenksj
and
claude
committed
0741f9d
fix(delta): nuke spark-warehouse on regression rerun + skip auto-flip on derived sessions

schenksj
and
claude
committed
533893c
fix(delta): add synthesizedFilePartitions + patch DeltaSinkSuite/IdentityColumnAdmission

schenksj
and
claude
committed
7b6e780
docs(delta): add end-to-end design + internals reference under docs/delta-integration/

schenksj
and
claude
committed
db5fe82
docs(delta): correct row-tracking config keys + document table-root URI double-encoding

schenksj
and
claude
committed
3b46792
fix(delta): decline native scan for CM-id mode + stale CM-name reads

schenksj
and
claude
committed
810011f
fix(delta): accept HTTP 3xx redirects in DeltaErrorsSuite URL validation

schenksj
and
claude
committed
be227f9
fix(delta): accept Comet's parquet error in SC-8810 corrupted-file test

schenksj
and
claude
committed
f993853
test(delta): skip 2B-row DV delete test in Comet regression diff

schenksj
and
claude
committed
b0ec773

Commits on May 13, 2026

perf(delta): cache no-arg Method handles in DeltaReflection lookups

schenksj
and
claude
committed
19fc6af
test(delta): bump test heap to 4g and add JDK 17 --add-opens flags

schenksj
and
claude
committed
5496bfb
docs(delta): capture #79 investigation notes in CometScanRule gate comment

schenksj
and
claude
committed
60c24ce
feat(delta): materialise DVs on pre-materialised FileIndex paths

schenksj
and
claude
committed
779eeef
docs(delta): inline TODOs for #75 input_file_name native support design A

schenksj
and
claude
committed
8588afe
feat(delta): keep input_file_name() native via one-task-per-partition

schenksj
and
claude
committed
a83ecb3

Commits on May 14, 2026