Skip to content

fix: OOM prevention with dynamic memory guard#65

Open
EVWorth wants to merge 4 commits intomainfrom
fix/oom-large-tables
Open

fix: OOM prevention with dynamic memory guard#65
EVWorth wants to merge 4 commits intomainfrom
fix/oom-large-tables

Conversation

@EVWorth
Copy link
Copy Markdown
Owner

@EVWorth EVWorth commented May 8, 2026

Summary

Replaces hardcoded 50k row limit with dynamic memory guard that monitors available system memory.

Changes

  1. Refactored MemoryGuard (3 commits)

    • Initial: Hard 50k cap → rustfmt fixes → dynamic guard
    • Monitors available system memory instead of per-process RSS
    • Triggers when 95% of available memory consumed (5% safety buffer)
  2. Key Implementation Details

    • Checks memory every 1000 rows during query execution
    • Early detection prevents OS OOM killer
    • Sets
      ows_truncated\ flag on result with hint message
    • Frontend virtualization handles large result sets safely
  3. Frontend Virtualization

    • @tanstack/react-virtual\ renders only visible rows
    • Works in tandem with backend limit enforcement

Testing

  • Frontend: 275 tests passing ✅
  • Rust clippy: No warnings ✅
  • Manual review: Memory guard logic verified

Risk Mitigation

  • Available memory threshold prevents false positives (stops too early) and false negatives (crashes anyway)
  • Graceful error handling with actionable user message
  • Dynamic threshold adapts to system load

EVWorth and others added 4 commits May 2, 2026 18:34
- Always apply LIMIT to SELECT/SHOW/DESCRIBE/EXPLAIN (was conditional)
- Hard cap at 50,000 rows regardless of user settings
- Fix rows_truncated detection to compare against capped_limit
- Lower frontend virtualization threshold from 500 to 100 rows
- Add truncation warning banner when results are capped

Closes #25

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Remove hardcoded 50k row cap
- Add sysinfo crate for process memory monitoring
- MemoryGuard checks RSS every 1000 rows during fetch
- Triggers when process hits 75% of system RAM
- Gracefully stops fetch and returns accumulated rows with warning
- Add OutOfMemory error variant to CoreError
- Update frontend to show backend warnings
- Truncation banner no longer mentions arbitrary 50k number
- Collapse nested if into single condition (clippy::collapsible_if)
- Use is_multiple_of(1000) instead of % 1000 == 0 (clippy::manual_is_multiple_of)
- Run cargo fmt for consistent formatting
- Changed MemoryGuard to monitor available system memory, not total
- Trigger threshold: 95% of available memory consumed (5% buffer)
- Previous logic used 75% of total memory which could allow crashes when other processes consumed RAM
- New logic prevents runaway queries when system memory is critically low
- Removed per-process RSS tracking; now monitors system-wide availability

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant