[SPARK-57268][SQL] Add Apache Arrow as a native cache format for in-memory Dataset caching#56334

Open

viirya wants to merge 37 commits into

apache:masterfrom

viirya:arrow-cache-format

Commits on Jun 4, 2026

Add Apache Arrow as native cache format for Spark in-memory Dataset caching

viirya
and
claude
committed
Implement min/max statistics collection for Arrow cache Phase 2
viirya
committed
Implement zero-copy optimization for ArrowColumnVector input
viirya
committed
Add comprehensive performance benchmarks for Arrow cache format
viirya
committed
Update Arrow cache benchmark with working implementation
viirya
committed
Add comprehensive Phase 3 documentation for Arrow cache format

viirya
and
claude
committed
Fix zero-copy benchmark and correct documentation about Parquet/ORC

viirya
and
claude
committed
Add Arrow cache benchmark with performance results

viirya
and
claude
committed
Update Arrow cache documentation with accurate benchmark results

viirya
and
claude
committed
Implement proper type checking for Arrow cache columnar input support

viirya
and
claude
committed
Add clarifying comments to ArrowUtils.isSupportedByArrow method

viirya
and
claude
committed
Remove unsupported claims about when default cache performs better

viirya
and
claude
committed
Add test to verify Arrow cache serializer is actually used

viirya
and
claude
committed
Optimize Arrow cache performance with three major improvements

viirya
and
claude
committed
[SPARK-XXXXX] Comment out LZ4 compression benchmarks in ArrowCacheBenchmark

viirya
and
claude
committed
[SPARK-XXXXX] Add ZSTD level -1 (fastest) compression benchmarks

viirya
and
claude
committed
[SPARK-XXXXX] Update ArrowCacheBenchmark results with ZSTD level -1

viirya
and
claude
committed
[ARROW-CACHE] Add column pruning benchmark (select 1 of 20 columns)

viirya
and
claude
committed
[ARROW-CACHE] Deduplicate methods in ArrowCachedBatchSerializer

viirya
and
claude
committed
[ARROW-CACHE] Add comprehensive tests for complex types from columnar input

viirya
and
claude
committed
[SPARK-XXXXX] Make ObjectColumnStats handle columnar complex types

viirya
and
claude
committed
[ARROW-CACHE] Optimize Arrow cache columnar-to-row read path
viirya
committed
[ARROW-CACHE] Add TPC-DS cache benchmark for Arrow vs Default comparison
viirya
committed
[ARROW-CACHE] Add columnar read and column pruning benchmarks to TPCDSCacheBenchmark
viirya
committed
[ARROW-CACHE] Inline statistics collection in columnar-to-Arrow write path
viirya
committed
[ARROW-CACHE] Fix collated strings, Kryo registration, NaN handling, and add tests
viirya
committed
[ARROW-CACHE] Add background prefetch for Arrow cache read path
viirya
committed
[ARROW-CACHE] Port minor improvements from merged PR review
viirya
committed
[ARROW-CACHE] Link Arrow cache docs into SQL menu and soften benchmark claims
viirya
committed
[ARROW-CACHE] Remove Arrow cache tuning guide doc
viirya
committed
[ARROW-CACHE] Remove Arrow cache migration guide doc
viirya
committed
[ARROW-CACHE] Complete the min/max statistics type list in docs
viirya
committed
[ARROW-CACHE] Remove TPCDSCacheBenchmark
viirya
committed
Benchmark results for org.apache.spark.sql.execution.benchmark.ArrowCacheBenchmark (JDK 25, Scala 2.13, split 1 of 1)
viirya
committed
Benchmark results for org.apache.spark.sql.execution.benchmark.ArrowCacheBenchmark (JDK 21, Scala 2.13, split 1 of 1)
viirya
committed
Benchmark results for org.apache.spark.sql.execution.benchmark.ArrowCacheBenchmark (JDK 17, Scala 2.13, split 1 of 1)
viirya
committed

Commits on Jun 5, 2026

[ARROW-CACHE] Fix CI failures: binding policy, test isolation, scalafmt
viirya
committed