Skip to content

Add disk-bounded cache with eviction and source fallback - #492

Open
cocosz wants to merge 3 commits into
datafusion-contrib:mainfrom
cocosz:fallback-reader
Open

cocosz wants to merge 3 commits into
datafusion-contrib:mainfrom
cocosz:fallback-reader

Conversation

@cocosz

@cocosz cocosz commented Apr 30, 2026

Copy link
Copy Markdown

Summary

Adds configurable disk size limits to LiquidCache with automatic eviction when the budget is exceeded, and a Parquet fallback reader so queries don't crash if a cache entry is evicted mid-read.

Changes

1. Parquet Fallback Reader

When ReadFromCache encounters a missing entry (evicted from cache), it reads the batch directly from the source Parquet file instead of erroring. This makes the read path resilient to concurrent eviction.

2. Disk Eviction

Configurable max_disk_bytes with a watermark (default 0.9). When disk usage exceeds max_disk_bytes * watermark, entries are evicted via the cache policy's find_disk_victims (FIFO among disk entries for LiquidPolicy).

3. Configuration

LiquidCacheLocalBuilder::new()
    .with_max_disk_bytes(10 * 1024 * 1024 * 1024) // 10 GB
    .with_disk_watermark(0.9)                      // evict at 90%
    .build(config)
    .await?;

How it works

Disk usage Behavior
Below watermark Write normally, no eviction
Above watermark Evict oldest disk entries until under budget, then write
Entry evicted mid-query Fallback reader fetches from Parquet directly

Testing

  • All existing tests pass
  • Added unit tests for: BudgetAccounting disk tracking, ArtIndex::remove, LiquidPolicy::find_disk_victims, fallback reader with evicted batch

@codacy-production

codacy-production Bot commented Apr 30, 2026

Copy link
Copy Markdown

Up to standards ✅

🟢 Issues 2 medium

Results:
2 new issues

Category Results
Complexity 2 medium

View in Codacy

🟢 Metrics 28 complexity · 10 duplication

Metric Results
Complexity 28
Duplication 10

View in Codacy

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

@cocosz cocosz changed the title Fallback reader Fall back to Parquet on cache miss during read Apr 30, 2026
@cocosz cocosz changed the title Fall back to Parquet on cache miss during read Fall back to source on cache miss during read Apr 30, 2026
@cocosz
cocosz force-pushed the fallback-reader branch from 1e96900 to 41ada25 Compare May 3, 2026 21:53
Tanvir Alam added 2 commits May 4, 2026 03:29
Signed-off-by: Tanvir Alam <tanvralm@amazon.com>
@cocosz
cocosz force-pushed the fallback-reader branch from a85e1b3 to 66a73c3 Compare May 3, 2026 22:23
@cocosz cocosz changed the title Fall back to source on cache miss during read Add disk-bounded cache with eviction and Parquet fallback May 3, 2026
@cocosz cocosz changed the title Add disk-bounded cache with eviction and Parquet fallback Add disk-bounded cache with eviction and source fallback May 3, 2026
Signed-off-by: Tanvir Alam <tanvralm@amazon.com>
@cocosz
cocosz force-pushed the fallback-reader branch from 66a73c3 to 19eef44 Compare May 3, 2026 22:36
@XiangpengHao

Copy link
Copy Markdown
Collaborator

Hi @cocosz Thank you for the contribution! This change is larger than I initially thought, and I feel it's easier to support this feature from my end. I've opened another pr #495 feel free to check out and let me know your thoughts!

@cocosz

cocosz commented May 5, 2026

Copy link
Copy Markdown
Author

Thanks @XiangpengHao! Really appreciate you taking this on — you know the codebase best and glad the discussion and exploration here was useful in shaping the direction. Will check out #495 and leave any thoughts there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants