duckdb: execute splits, not partitions - #9278
Conversation
Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
Merging this PR will improve performance by 12.44%
Performance Changes
Tip Curious why this is faster? Comment Comparing Footnotes
|
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Vortex (geomean): 0.952x ➖ datafusion / vortex-file-compressed / ns (0.952x ➖, 3↑ 0↓)
File Size Changes (1 files changed, +0.0% overall, 1↑ 0↓)
Totals:
|
Benchmarks: Statistical and Population Genetics 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
duckdb / vortex-file-compressed / ns (1.094x ➖, 1↑ 5↓)
duckdb / parquet / ns (1.002x ➖, 0↑ 0↓)
File Size Changes (2 files changed, -32.3% overall, 0↑ 2↓)
Totals:
|
Benchmarks: TPC-H SF=1 on S3 📖Verdict: No clear signal (environment too noisy confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.156x ➖, 0↑ 4↓)
datafusion / parquet / ns (1.036x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed / ns (0.968x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.995x ➖, 0↑ 0↓)
|
Benchmarks: Clickbench Sorted on NVME 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.893x ✅, 3↑ 0↓)
datafusion / parquet / ns (0.950x ➖, 1↑ 0↓)
duckdb / vortex-file-compressed / ns (0.975x ➖, 1↑ 0↓)
duckdb / parquet / ns (0.985x ➖, 0↑ 0↓)
File Size Changes (201 files changed, -42.8% overall, 51↑ 150↓)
Totals:
|
Benchmarks: FineWeb S3 📖Verdict: No clear signal (environment too noisy confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.273x ➖, 0↑ 4↓)
datafusion / parquet / ns (0.936x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.023x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.522x ❌, 0↑ 2↓)
|
Benchmarks: FineWeb NVMe 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.032x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.994x ➖, 1↑ 0↓)
duckdb / vortex-file-compressed / ns (1.026x ➖, 0↑ 1↓)
duckdb / parquet / ns (0.995x ➖, 0↑ 0↓)
File Size Changes (2 files changed, -46.3% overall, 0↑ 2↓)
Totals:
|
Benchmarks: TPC-H SF=10 on NVME 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.991x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.015x ➖, 0↑ 1↓)
duckdb / parquet / ns (1.013x ➖, 0↑ 0↓)
File Size Changes (9 files changed, -44.0% overall, 0↑ 9↓)
Totals:
|
Benchmarks: TPC-DS SF=1 on NVME 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.998x ➖, 1↑ 2↓)
datafusion / parquet / ns (1.003x ➖, 2↑ 2↓)
duckdb / vortex-file-compressed / ns (1.045x ➖, 1↑ 23↓)
duckdb / parquet / ns (1.001x ➖, 2↑ 5↓)
File Size Changes (25 files changed, -43.5% overall, 0↑ 25↓)
Totals:
|
Benchmarks: TPC-H SF=1 on NVME 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.014x ➖, 0↑ 2↓)
duckdb / vortex-file-compressed / ns (1.015x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.001x ➖, 0↑ 0↓)
File Size Changes (9 files changed, -43.9% overall, 0↑ 9↓)
Totals:
|
Benchmarks: Clickbench on NVME 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.985x ➖, 1↑ 0↓)
datafusion / parquet / ns (1.006x ➖, 0↑ 2↓)
duckdb / vortex-file-compressed / ns (0.998x ➖, 4↑ 1↓)
duckdb / parquet / ns (1.006x ➖, 1↑ 1↓)
File Size Changes (101 files changed, -39.2% overall, 0↑ 101↓)
Totals:
|
TODO