feat: add LanceDB version snapshot auto-cleanup

[njuboy11]

Summary

LanceDB stores a new version snapshot on every write operation but never cleans them up. After a few days the database directory grows to 300+ MB despite only ~5-30 MB of actual data.

This PR adds a storageMaintenance.autoCleanup config block that periodically calls Table.optimize() to remove old version snapshots.

Config

{
  "storageMaintenance": {
    "autoCleanup": {
      "enabled": true,
      "intervalHours": 24,
      "retentionDays": 1
    }
  }
}

Changes

• Add storageMaintenance.autoCleanup to plugin config schema (enabled, intervalHours, retentionDays)
• Add MemoryStore.runMaintenance(retentionDays) → calls table.optimize({ cleanupOlderThan })
• Add non-blocking setInterval timer in registerService.start (fires 5 min after startup, then every intervalHours)
• Latest version is NEVER deleted by LanceDB

Closes #792

[TurboTheTurtle]

Review findings:

P1: maintenanceTimer is scoped inside start() but used in stop()

maintenanceTimer is declared inside the start callback, but stop is a sibling callback and references that variable later (index.ts:4176, index.ts:4210). That variable is out of scope for stop, so this either fails type checking or throws when the service stops. Please hoist maintenanceTimer to the same scope as backupTimer so start and stop share the same timer handle.

P1: accidental /tmp/rerank.log write leaks recall queries

The auto-recall path now does a synchronous append to /tmp/rerank.log with the recall query text (index.ts:2596). That leaks user memory queries to a fixed temp file and adds sync filesystem I/O on a hot recall path. This looks like debug instrumentation and should be removed before merge.

P1: lock ENOENT falls back to lock-free writes

runWithFileLock() now catches ENOENT from proper-lockfile.lock() and proceeds with the write without acquiring any cross-process lock (src/store.ts:242-272). That breaks the serialization guarantee around LanceDB writes, and it is unrelated to the storage-cleanup feature. If lock acquisition fails, the write should fail or retry under the lock rather than entering the critical section lock-free.

[rwmjhb]

PR #793 Review: feat: add LanceDB version snapshot auto-cleanup

Verdict: RESOLVE-CONFLICTS-FIRST | Author: njuboy11 | Merge state: DIRTY

Pipeline short-circuited at the conflict gate after R0 verification. Deep review deferred until the branch rebases cleanly onto the base.

Problem Statement (R1)

LanceDB creates version snapshots on writes and the plugin does not currently clean them up, so active installations can see the database directory grow far beyond the live data size over days of use. The PR attempts to add configurable periodic cleanup via LanceDB table optimization.

Why This Stopped Here

GitHub reports mergeable=CONFLICTING (merge_state_status=DIRTY) for this PR. Reviewing the diff now would:

1. Give feedback against a branch the author must rewrite anyway
2. Produce findings that may be invalidated by the conflict resolution
3. Waste review cycles on code that cannot be merged as-is

Recommended Action

Author should:

1. Rebase this branch onto the latest base (or merge the base into this branch)
2. Resolve all merge conflicts
3. Push the rebased branch — the re-review will be picked up automatically

---

Reviewed at 2026-05-17T08:54:39Z | R0+R1 gate | Conflict gate

[rwmjhb]

Requesting changes. This PR has several P1 issues that need to be fixed before merge, independent of the current merge conflict.

• maintenanceTimer appears to be scoped inside start() but referenced from sibling stop(). Please hoist it to shared plugin scope so both lifecycle callbacks use the same handle.
• The auto-recall path writes recall query text to /tmp/rerank.log. That looks like debug instrumentation, leaks user memory queries to a fixed temp path, and adds sync I/O on a hot path. Please remove it.
• runWithFileLock() appears to catch ENOENT from lock acquisition and continue into the write path without a cross-process lock. That weakens the LanceDB write serialization guarantee and is unrelated to storage cleanup. Please fail/retry under lock rather than performing lock-free writes.

After fixing, please rebase onto latest origin/master, preserve all current scripts/ci-test-manifest.mjs entries, and run:

npm run test:packaging-and-workflow
npm run test:core-regression

[rwmjhb]

PR #793 Review: feat: add LanceDB version snapshot auto-cleanup

Verdict: RESOLVE-CONFLICTS-FIRST | Author: njuboy11 | Merge state: DIRTY

Pipeline short-circuited at the conflict gate after R0 verification. Deep review deferred until the branch rebases cleanly onto the base.

Problem Statement (R1)

LanceDB write operations create version snapshots that the plugin never cleans up, causing active database directories to grow far beyond the live data size over days. The PR proposes configurable periodic LanceDB table optimization to reclaim old snapshots.

Why This Stopped Here

GitHub reports mergeable=CONFLICTING (merge_state_status=DIRTY) for this PR. Reviewing the diff now would:

1. Give feedback against a branch the author must rewrite anyway
2. Produce findings that may be invalidated by the conflict resolution
3. Waste review cycles on code that cannot be merged as-is

Recommended Action

Author should:

1. Rebase this branch onto the latest base (or merge the base into this branch)
2. Resolve all merge conflicts
3. Push the rebased branch — the re-review will be picked up automatically

---

Reviewed at 2026-05-23T08:59:36Z | R0+R1 gate | Conflict gate