feat(indexing): reuse embeddings, batch large reindexes, prune stale chunks

[julianam-w]

Summary

• Embedding reuse: every stored chunk carries a SHA-256 content_hash. Ingest, sync and reindex look up an existing embedding by hash before calling Voyage, so unchanged text is never re-embedded, and identical texts within one run are embedded once. Incremental sync/reindex now embed before deleting anything and swap old chunks for new in a single transaction, so a Voyage failure leaves the index as it was.
• Voyage Batch API for large full reindexes: a full reindex (no baseline SHA, a diff too large for GitHub's compare API, or ingest.py --batch) with at least VOYAGE_BATCH_THRESHOLD (default 2000) new texts submits them to the Voyage Batch API (33% cheaper, up to 12h) instead of embedding in real time. A submitting sync run leaves the namespace as it is; later runs skip it while the batch is in flight, then load the results into rag_embedding_cache and finish the reindex from the cache. Incremental syncs always stay real time.
• Index pruning: a full reindex now deletes chunks the checkout no longer produces (removed/newly-skipped files, truncated chunk ranges) once every upsert has succeeded, and refuses to run on a checkout with no indexable files.
• Hourly sync: documented the cron as 0 * * * * instead of daily — each run only embeds changed text, a namespace with nothing new costs one GitHub API call, and Railway skips a run while the previous one is still going.
• Spec: specs/pipelines/embedding-sync.md, status implemented, 18 BL clauses / 17 ACs.

Linear / spec

• No Linear issue.
• Spec: specs/pipelines/embedding-sync.md

Test plan

• Spec conformance: all 18 BL clauses checked against the code clause by clause; all 17 ACs have a passing automated test (uv run pytest, 27 passed — 3 require TEST_DATABASE_URL, which the suite skips cleanly without it).
• python .maui/scripts/check_spec_anchors.py / check_bl_length.py: all clauses anchored, all within two sentences.
• /code-review --fix (8-angle parallel review): found and fixed 3 real bugs —
  1. delete_chunks_except(conn, table, []) wiped the whole table instead of deleting nothing (execute_values silently skips an empty INSERT, leaving the "keep" set empty so the anti-join matched everything). Would have silently emptied a namespace's docs (or code) table on any full reindex where that table produced zero chunks. Reproduced and fixed against a real Postgres instance.
  2. batch_threshold() raised SystemExit, which sync.py's per-namespace except Exception doesn't catch — one bad VOYAGE_BATCH_THRESHOLD would have aborted the whole cron run instead of failing just that namespace. Now raises ValueError.
  3. The batch-collection branch of sync_namespace advanced state without checking whether its own reindex succeeded, unlike the other two full-reindex call sites. Hardened to match (unreachable as a live bug today, closes a gap for future changes).
• Live end-to-end, against the real database and Voyage API (patient-dedupe-matching):
  • ingest.py --repo ... --batch below the default threshold → real-time embed, 567 chunks, namespace registered at its SHA (AC-001).
  • ingest.py --repo ... --batch with VOYAGE_BATCH_THRESHOLD=1 against a fresh namespace → submitted one real batch, no real-time embedding, namespace registered with no SHA (AC-002).
  • sync.py collected the completed batch, cached all 567 output embeddings, then ran a full reindex that reused 100% of them from the cache with zero real-time Voyage calls, and advanced the SHA to the repo's current HEAD (AC-005, AC-008).

Risk / rollback

• A namespace's existing tables get a one-off content_hash backfill the first time they're set up again after this merges.
• OQ-001 (spec): Voyage doesn't document a max batch upload file size; 100 MB per batch is a conservative, untested assumption — the live test only exercised a 567-line file.
• Follow-up outside this PR: the Railway cron schedule still needs changing to 0 * * * * in the dashboard (not stored in the repo).
• Rollback: revert the merge commit. No destructive migration — the new tables (rag_pending_batches, rag_embedding_cache) and column (content_hash) are additive.

Checklist

• CI green
• Docs updated if applicable
• Spec status updated if implementing/changing a spec
• Spec change-log squashed to a single merge-dated entry; resolved DV-XXX / OQ-XXX rows deleted (not annotated) — OQ-001 remains open (unresolved, not resolved)

🤖 Generated with Claude Code

Update: second /code-review --fix pass

After pushing, I re-ran /code-review --fix (8-angle review) against the updated branch. All findings were consolidated and verified before fixing. Four more real issues, fixed:

• setup_table() ran its column/backfill/index migration unconditionally on every call. ALTER TABLE takes its lock before checking IF NOT EXISTS, so with sync now hourly this repeated a lock acquisition against a live table the MCP server concurrently reads, on every tick, forever, for no effect after the first run. Gated behind an information_schema existence check.
• ingest() never closed its database connection if embedding, batch submission or pruning raised — a real connection leak, worse now that sync.py calls it in a per-namespace loop. Wrapped the function body in try/finally.
• get_embeddings_by_hash's DISTINCT ON had no ORDER BY, making its tie-break between duplicate rows for a hash plan-dependent. Added a deterministic ORDER BY.
• register.py --sha "" (documented as forcing a full reindex) didn't clear a namespace's pending Voyage batch, so a stuck or in-flight batch would silently override it — sync.py checks pending batches before ever looking at last_commit_sha. Now also clears pending batch rows and cached embeddings, giving operators a supported recovery path for a batch that can never resolve.

Also restored per-file delete logging and flush=True on progress prints (both dropped by the reindex/sync refactor, useful for diagnosing a stuck job against Railway's piped logs), and fixed three test names and one module docstring to match the team's conventions.

Adds BL-019..021 / AC-018..020 to the spec. Full suite: 33 passed (30 + 3 Postgres-gated, which also ran this time).

Considered, not fixed: a claimed Voyage expired batch status (not in Voyage's documented status set, unlike OpenAI's Batch API — checked against their docs, not acted on); several reuse/duplication suggestions (shared SQL/HTTP helpers, a shared batching helper) deferred as broad refactors rather than bugs.