Skip to content

feat(persistence): Compact release databases - #88

Merged
hubab1 merged 6 commits into
mainfrom
codex/release-storage-compaction
Oct 26, 2026
Merged

hubab1 merged 6 commits into
mainfrom
codex/release-storage-compaction

Conversation

@hubab1

@hubab1 hubab1 commented Oct 24, 2026 •

Copy link
Copy Markdown
Owner

What

  • Migrate existing ratings once into validated archives and reclaim unused space, with resumable recovery and visible upgrade progress.
  • Keep ingestion compact and preserve history on provider misses or storefront mismatches. Bound MCP, export, and sparkline reads under shared writer coordination.
  • All 738 tests pass locally and in CI. Production-copy validation measures 1.979 → 1.322 GiB with every rating and other research data unchanged. This is 3.10× below the original 4.099 GiB baseline.

Why

Repeated ratings and internal framework changes inflated release databases. Automatic storefront cleanup also erased valid history, while small history responses decoded lifetime data. Compact ingestion and coordinated reads preserve historical dates and values without recurring archival scans. Merge #87 first.

Archive complete daily rating records in bounded compressed groups.
Validate saved payloads before removing source rows, and resume from
durable checkpoints after interruptions.

Prune unused internal history and compact stores during coordinated
release startup. Keep charts, exports, MCP reads, and corrections able
to access every historical observation.
Repack large stores with 16 KiB pages and prefetch live rating links.
Prepare persistence off the UI thread and show startup progress.
Verify that history survives compaction and WAL reopening.
@hubab1
hubab1 force-pushed the codex/release-storage-compaction branch from 0995441 to 20f8ca4 Compare October 24, 2026 22:36
@hubab1
hubab1 marked this pull request as ready for review October 24, 2026 22:37
Archive prior days during ingestion and update older days in place.
Keep the existing-data migration complete after its first successful run.
Serialize worker transactions to preserve concurrent archive edits.
Reject shared clients until migration finishes and validate every saved
archive before replacing it.

Defer disk-full compaction without hiding corruption errors.
Keep the pending flag so a later launch can retry safely.
Recognize SQLite error codes carried in Core Data error metadata.
Keep corruption fatal and preserve history during a compaction retry.
Keep past ratings when providers report missing or mismatched storefronts.
Make explicit clearing atomic across live rows, archives, and summaries.

Read only required history chunks under a shared writer lease.
Defer unused archive fetches and expose migration progress and read errors.
@hubab1
hubab1 changed the base branch from codex/keyword-navigation-performance to main October 26, 2026 21:22
@hubab1
hubab1 merged commit 932eea6 into main Oct 26, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant