Bug: readRecentCandidates() loads entire file into memory causing OOM crash

[zhongliangyu]

Problem

The readRecentCandidates() function in src/gep/assetStore.js reads the entire candidates.jsonl file into memory, even though it only needs the last N lines:

function readRecentCandidates(limit = 20) {
  const raw = fs.readFileSync(p, 'utf8');  // Reads ENTIRE file!
  const lines = raw.split('\n').map(l => l.trim()).filter(Boolean);
  return lines.slice(Math.max(0, lines.length - limit))...
}

Impact

When candidates.jsonl grows large (my file reached 83MB), this causes:

• Memory spike during each evolution cycle
• Potential OOM crash when combined with other memory usage
• Process restart/killed by the suicide check (EVOLVER_MAX_RSS_MB)

Reproduction

1. Run evolver for an extended period (weeks/months)
2. candidates.jsonl accumulates to tens of MB
3. Observe memory usage spike or process crash

Suggested Fix

Use streaming or tail approach to only read the last N lines:

function readRecentCandidates(limit = 20) {
  const { execSync } = require('child_process');
  try {
    const output = execSync(`tail -n ${limit} "${p}"`, { encoding: 'utf8', timeout: 5000 });
    return output.split('\n').filter(Boolean).map(l => {
      try { return JSON.parse(l); } catch { return null; }
    }).filter(Boolean);
  } catch { return []; }
}

Or implement a proper backwards file reader for Windows compatibility.

Environment

• evolver version: v1.27.5
• Node.js: v20.x
• OS: Linux
• candidates.jsonl size: 83MB (before cleanup)

Thanks!

[zhongliangyu]

Update: Found the actual root cause

After further investigation, I discovered that memory_graph.jsonl was the actual culprit (1.5GB!), not candidates.jsonl (which was 83MB).

The same issue exists in src/gep/memoryGraph.js:119:

function tryReadMemoryGraphEvents(limitLines = 2000) {
  const raw = fs.readFileSync(p, 'utf8');  // Reads ENTIRE 1.5GB file!
  const lines = raw.split('\n');
  const recent = lines.slice(Math.max(0, lines.length - limitLines));  // Only needs last 2000 lines
  // ...
}

Memory consumption for 1.5GB file:

• Raw string: ~1.5GB
• Array after split: ~2-3GB
• Total: 3-4GB+

This exceeds the EVOLVER_MAX_RSS_MB=2048 limit and causes OOM/crash.

Suggested fix: Both readRecentCandidates() and tryReadMemoryGraphEvents() should use streaming or tail approach to only read the last N lines instead of loading entire files into memory.

[autogame-17]

Thank you @zhongliangyu for this thorough bug report and the follow-up analysis on memory_graph.jsonl. Both are valid issues.

Reading multi-MB (or GB) files into memory just to extract the last N lines is clearly problematic for long-running nodes. We will address both readRecentCandidates() and tryReadMemoryGraphEvents() to use a streaming/tail approach instead of loading entire files.

This is tracked internally and will be included in a future release.