Skip to content

Bound local analytics memory with a derived SQLite store #36

Description

@thrr87

Problem

Codex Limits still retains local analytics history as full Swift arrays.
The latest bounded restore lowers the first refresh, but each later refresh
adds more facts to the resident set.

Measured on the current local dataset:

  • 406,727 Derived Records and about 400 MB of app-owned JSONL
  • installed pre-fix build: 1.62–2.36 GB peak RSS on launch
  • current codex/reduce-refresh-cost build: 122–135 MB on the first refresh
  • 20,000-record restore: +38.6 MB RSS
  • 100,000-record restore: +266.5 MB RSS

The current seam publishes [LocalActivityFact]. UsageMonitor retains that
history and the reader builds receipts, diagnostics, timelines, and indexes
from it. Bounded reading delays the growth but cannot bound the final resident
set.

A native SQLite probe over all 406,727 records used:

  • 28.4 MB peak RSS during a 4.31 s indexed import
  • 178.5 MB on disk
  • 22.5 MB peak RSS for a fresh query process

These figures validate SQLite as a local derived store. They do not set final
product timings.

Direction

Create one deep LocalActivityStore module backed by system SQLite3.

The module must own incremental ingest, source cursors, normalization,
deduplication, local projections, and bounded queries. It must return summaries,
series, and pages. It must never return the full fact history.

Keep:

  • SwiftUI and the current Swift app
  • Codex source files read-only
  • account usage history and optional folder sync in the current JSON format
  • Coverage, Confidence, Task Tree, Project, and deletion rules

Do not add Rust, a resident helper, an ORM, a connection pool, or a new package.

Delivery

  1. Remove the full refresh tied to opening the menu. Show the last account
    snapshot at once; keep launch, ten-minute, wake, and manual refresh paths.
  2. Add a performance contract and parity fixtures for the current aggregators.
  3. Add a local, versioned SQLite store and import legacy Derived Records in
    bounded transactions.
  4. Move local token, timeline, filter, receipt, and task-detail reads behind
    bounded store queries.
  5. Remove [LocalActivityFact] from the monitor/reader seam and retire the old
    JSONL cache only after a verified migration.

Required behavior

  • The default Usage remaining graph performs no local-store or Codex-source read.
  • Import reads Codex sources without changing them.
  • One transaction commits Derived Records and its source cursor together.
  • Cancellation never publishes a partial batch.
  • Queries see one committed revision and return bounded results.
  • Receipt and Task Tree detail use stable keyset pages.
  • Missing source data lowers Coverage and names the reason.
  • Delete analytics history removes the database, its helper files, legacy
    local cache, account history, and sync-folder history.
  • Automatic import cannot restore data older than the deletion cutoff.
  • Rebuild remains a separate user action.
  • A failed migration keeps the old cache and reports the failure.

Performance gate

Validate on a copy of the current real dataset:

  • default menu and Usage remaining: at most 120 MB RSS
  • an ordinary loaded local view: at most 150 MB RSS
  • import peak: at most 250 MB RSS
  • no upward RSS trend across ten no-change refreshes
  • menu open: at most 50 ms on the main thread
  • bounded local read: at most 150 ms cold

If the spike misses these gates, stop and profile before completing the
migration. Consider Rust only if parsing remains the measured bottleneck after
full-history residency is gone.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions