Skip to content

clean metadata cache - #519

Merged
XiangpengHao merged 1 commit into
mainfrom
metadata-cache-clean
Sep 4, 2026
Merged

XiangpengHao merged 1 commit into
mainfrom
metadata-cache-clean

Conversation

@XiangpengHao

Copy link
Copy Markdown
Collaborator

Latest data fusion already has metadata cache, so we don't need to do it again.

@codacy-production

Copy link
Copy Markdown

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics -8 complexity · 4 duplication

Metric Results
Complexity -8
Duplication 4

View in Codacy

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

@codecov

codecov Bot commented Sep 4, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 92.79279% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.54%. Comparing base (7c4933f) to head (fd30a33).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
src/datafusion/src/reader/plantime/morselizer.rs 95.12% 0 Missing and 4 partials ⚠️
src/datafusion/src/cache/stats.rs 0.00% 0 Missing and 2 partials ⚠️
...tafusion/src/reader/runtime/liquid_cache_reader.rs 88.88% 0 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #519      +/-   ##
==========================================
+ Coverage   82.49%   82.54%   +0.04%     
==========================================
  Files          66       66              
  Lines       11872    11799      -73     
  Branches    11872    11799      -73     
==========================================
- Hits         9794     9739      -55     
+ Misses       1830     1812      -18     
  Partials      248      248              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@XiangpengHao
XiangpengHao merged commit 328e176 into main Sep 4, 2026
15 checks passed
@github-actions

github-actions Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

📊 Benchmark Comparison

Current: 720f7ccf (Liquid) vs Baseline: 720f7ccf (DataFusionDefault)

Query Cold Time Δ Warm Time Δ CPU Time Δ
Q1 3.0ms (3.0ms) +0.0% 0.000ms (0.000ms) +0.0% 0.000ms (0.000ms) +0.0%
Q2 11.0ms (7.0ms) +57.1% 6.5ms (6.0ms) +8.3% 7.5ms (6.0ms) +25.0%
Q3 17.0ms (14.0ms) +21.4% 6.0ms (11.5ms) -47.8% 4.5ms (22.5ms) -80.0%
Q4 17.0ms (12.0ms) +41.7% 4.0ms (11.0ms) -63.6% 3.5ms (24.0ms) -85.4%
Q5 66.0ms (47.0ms) +40.4% 52.0ms (51.0ms) +2.0% 5.5ms (25.5ms) -78.4%
Q6 245.0ms (100.0ms) +145.0% 87.0ms (102.0ms) -14.7% 29.5ms (79.5ms) -62.9%
Q7 1.0ms (1.0ms) +0.0% 1.0ms (1.0ms) +0.0% 0.000ms (0.000ms) +0.0%
Q8 11.0ms (6.0ms) +83.3% 6.0ms (5.5ms) +9.1% 9.5ms (6.0ms) +58.3%
Q9 111.0ms (88.0ms) +26.1% 84.5ms (82.5ms) +2.4% 5.0ms (43.0ms) -88.4%
Q10 107.0ms (88.0ms) +21.6% 85.5ms (84.0ms) +1.8% 9.0ms (60.5ms) -85.1%
Q11 56.0ms (26.0ms) +115.4% 24.0ms (25.0ms) -4.0% 44.5ms (36.0ms) +23.6%
Q12 54.0ms (28.0ms) +92.9% 25.0ms (29.0ms) -13.8% 44.0ms (45.5ms) -3.3%
Q13 212.0ms (112.0ms) +89.3% 101.5ms (106.0ms) -4.2% 59.0ms (81.5ms) -27.6%
Q14 589.0ms (131.0ms) +349.6% 138.0ms (134.0ms) +3.0% 83.0ms (104.5ms) -20.6%
Q15 392.0ms (100.0ms) +292.0% 109.0ms (98.5ms) +10.7% 75.0ms (93.5ms) -19.8%
Q16 125.0ms (95.0ms) +31.6% 115.5ms (97.5ms) +18.5% 4.0ms (25.0ms) -84.0%
Q17 688.0ms (204.0ms) +237.3% 257.0ms (208.5ms) +23.3% 97.0ms (104.0ms) -6.7%
Q18 645.0ms (203.0ms) +217.7% 221.0ms (204.0ms) +8.3% 69.0ms (103.5ms) -33.3%
Q19 920.0ms (434.0ms) +112.0% 380.0ms (382.0ms) -0.5% 95.0ms (151.5ms) -37.3%
Q20 16.0ms (12.0ms) +33.3% 3.5ms (12.0ms) -70.8% 8.0ms (23.5ms) -66.0%
Q21 613.0ms (170.0ms) +260.6% 469.5ms (174.5ms) +169.1% 307.0ms (270.0ms) +13.7%
Q22 1.01s (173.0ms) +483.8% 602.0ms (164.0ms) +267.1% 174.5ms (337.5ms) -48.3%
Q23 2.00s (451.0ms) +343.2% 1.67s (460.0ms) +263.6% 530.0ms (727.5ms) -27.1%
Q24 25.09s (865.0ms) +2800.9% 988.5ms (896.0ms) +10.3% 596.5ms (2.48s) -75.9%
Q25 295.0ms (72.0ms) +309.7% 17.0ms (58.5ms) -70.9% 46.5ms (120.0ms) -61.3%
Q26 129.0ms (47.0ms) +174.5% 19.5ms (46.5ms) -58.1% 51.5ms (82.5ms) -37.6%
Q27 291.0ms (59.0ms) +393.2% 25.0ms (60.0ms) -58.3% 75.0ms (119.0ms) -37.0%
Q28 789.0ms (214.0ms) +268.7% 945.5ms (217.5ms) +334.7% 218.0ms (268.0ms) -18.7%
Q29 1.45s (997.0ms) +44.9% 944.5ms (993.5ms) -4.9% 385.5ms (348.0ms) +10.8%
Q30 32.0ms (27.0ms) +18.5% 24.0ms (26.0ms) -7.7% 6.0ms (21.0ms) -71.4%
Q31 623.0ms (110.0ms) +466.4% 78.0ms (99.5ms) -21.6% 73.5ms (140.0ms) -47.5%
Q32 946.0ms (92.0ms) +928.3% 114.5ms (93.5ms) +22.5% 105.5ms (142.5ms) -26.0%
Q33 319.0ms (311.0ms) +2.6% 295.0ms (305.5ms) -3.4% 8.0ms (70.5ms) -88.7%
Q34 835.0ms (381.0ms) +119.2% 470.5ms (382.5ms) +23.0% 134.5ms (267.0ms) -49.6%
Q35 806.0ms (379.0ms) +112.7% 467.0ms (388.0ms) +20.4% 129.5ms (263.5ms) -50.9%
Q36 93.0ms (97.0ms) -4.1% 83.5ms (97.0ms) -13.9% 4.0ms (24.0ms) -83.3%
Q37 304.0ms (99.0ms) +207.1% 76.5ms (95.5ms) -19.9% 31.0ms (70.5ms) -56.0%
Q38 76.0ms (42.0ms) +81.0% 30.0ms (42.5ms) -29.4% 20.5ms (24.0ms) -14.6%
Q39 281.0ms (49.0ms) +473.5% 24.0ms (45.5ms) -47.3% 18.5ms (69.5ms) -73.4%
Q40 700.0ms (179.0ms) +291.1% 209.0ms (179.5ms) +16.4% 58.5ms (126.5ms) -53.8%
Q41 22.0ms (21.0ms) +4.8% 11.0ms (19.5ms) -43.6% 7.0ms (17.5ms) -60.0%
Q42 23.0ms (18.0ms) +27.8% 9.5ms (18.5ms) -48.6% 6.5ms (15.5ms) -58.1%
Q43 17.0ms (14.0ms) +21.4% 11.0ms (14.5ms) -24.1% 7.0ms (10.0ms) -30.0%

⚠️ LiquidCache is slower on 10 queries (warm)

  • Q28: warm +334.7% (945.5ms vs 217.5ms)
  • Q22: warm +267.1% (602.0ms vs 164.0ms)
  • Q23: warm +263.6% (1.67s vs 460.0ms)
  • Q21: warm +169.1% (469.5ms vs 174.5ms)
  • Q17: warm +23.3% (257.0ms vs 208.5ms)
  • Q34: warm +23.0% (470.5ms vs 382.5ms)
  • Q32: warm +22.5% (114.5ms vs 93.5ms)
  • Q35: warm +20.4% (467.0ms vs 388.0ms)
  • Q16: warm +18.5% (115.5ms vs 97.5ms)
  • Q40: warm +16.4% (209.0ms vs 179.5ms)

Compared Liquid vs DataFusionDefault on the same runner
Regressions: warm-time increases of at least 15%. Cold Time: first iteration; Warm Time: median of remaining iterations.

@XiangpengHao
XiangpengHao deleted the metadata-cache-clean branch September 17, 2026 18:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant