[GLUTEN][VL] Read Delta deletion vectors through Velox

[malinjawi]

What changes were proposed in this pull request?

This is PR B of the Delta deletion-vector performance work and is stacked on #12836.

• PR A ([GLUTEN][VL] Defer Delta deletion vector reads to executors #12836) fixes the regression by moving on-disk DV materialization from serial driver planning to Spark executors. That is the default path.
• This PR adds a separately reviewable Velox range-read path, disabled by default.

With spark.gluten.sql.columnar.delta.dv.nativeRangeRead.enabled=true, the JVM resolves the authoritative absolute DV URI and passes {absolute path, offset, payload size} to Velox. Velox reads that range through FileHandleFactory and BufferedInput, allowing DV I/O to use existing split preloading and async data caching. Inline DVs remain on the JVM byte-handoff path because their bytes are already present in Delta metadata.

The Substrait change is additive: the existing serialized-byte field remains supported. Before applying a bitmap, the native reader validates the range, Delta length prefix, CRC checksum, Roaring payload, and cardinality.

Until #12836 merges, GitHub shows both stacked commits in this PR. The PR B-specific change is commit 0ed763228.

Why are the changes needed?

#12836 removes the approximately 100-second serial driver barrier observed with 2,461 DV files on S3. This optional follow-up removes the remaining executor-JVM materialization and JNI payload copy, and lets Velox overlap and cache DV I/O.

Performance validation

The matched benchmark used Spark 3.5.4, fixed resources, identical queries, balanced arm ordering, and the same immutable Delta snapshot. The validation build retained a temporary legacy-driver control; the final #12836 code does not expose that rollback path.

Arm	Physical-scan median	vs legacy	All three queries
Legacy driver control	239.467 s	—	247.583 s
PR A executor-JVM read	19.085 s	12.55x	27.008 s
PR B Velox range read	13.907 s	17.22x	21.370 s

Query	Legacy	PR A	PR B	PR B vs PR A
Metadata count(*)	8.116 s	7.923 s	7.487 s	1.06x
Physical sum	123.592 s	12.332 s	9.027 s	1.37x
Physical grouped sum	114.912 s	6.753 s	4.880 s	1.38x

PR B improves the combined physical-scan median by 1.37x over PR A, from 19.085 s to 13.907 s. The combined median is calculated from each run's sum + grouped_sum time and therefore need not equal the sum of the displayed per-query medians.

The snapshot contained 2,461 data files, all with DVs: 7,199,920,789 rows before DV application, DV cardinality 720,032,919, and 6,479,887,870 visible rows. All timed applications returned identical results.

A separate metrics run recorded 4,922 native range reads, exactly 2 physical scans × 2,461 DV files, and approximately 623.3 MiB of DV payload. Its 282.4 executor-seconds are summed parallel task time, not wall time; the application completed in approximately 20.84 s and was excluded from timing medians.

A separate balanced three-round ingestion matrix over all 24 TPC-DS SF2500 tables produced these medians:

Arm	Full ingestion	DV-bearing store_sales
Legacy	820.56 s	372.35 s
PR A	693.24 s	254.08 s
PR B	695.28 s	255.66 s

Only store_sales carried DVs. PR A and PR B are effectively tied for full ingestion because writing 6.48 billion visible rows and processing 23 non-DV inputs dominate that workload. All nine applications completed 24 of 24 tables with matching schemas, partition columns, Delta operation metrics, row statistics, and output row/file counts.

The supported claim is narrow: #12836 fixes the original driver-side regression; this PR adds a further 1.37x improvement for the isolated physical DV scans. It does not claim a material PR B advantage over PR A for full ingestion, an improvement to DV creation or native writing, or Spark 4 performance.

How was this patch tested?

• ./dev/format-scala-code.sh
• clang-format 15 on every changed C++ file
• ./dev/gen-all-config-docs.sh
• Spark 3.3 / Scala 2.12 / Delta 2.3 test compilation
• Spark 3.5 / Scala 2.12 / Delta 3.3 test compilation
• Spark 4.0 / Scala 2.13 / Delta 4.0 test compilation
• Spotless, Scalastyle, Checkstyle, and git diff --check
• JVM descriptor-handoff, native range validation, split conversion, runtime-config, and cross-Delta-version tests

Linux native CI remains authoritative for the C++ build and tests.

Generative AI tooling

Generated-by: IBM BOB

[malinjawi]

Run Gluten Clickhouse CI

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[malinjawi]

Run Gluten Clickhouse CI

[malinjawi]

Run Gluten Clickhouse CI

[github-actions]

Run Gluten Clickhouse CI on x86

[malinjawi]

Run Gluten Clickhouse CI

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86