Persistent shape catalog + lazy restart recovery

[balegas]

Foundation for restart-safety (closes the root cause behind #3 and #4) and prerequisite for HTTP caching and retention work.

Scope (aligned in discussion):

• Catalog: persist {shapeId, table, canonical predicate, columns, kind, backfill-complete, tailer offset, last_read}. Proposed store: a __meta/shapes durable stream — keeps the engine node-stateless (all durable state = PG slot + DS log); boot folds the catalog stream only. last_read flushed periodically/batched, never per-read.
• Lazy recovery — boot must be O(catalog), not O(shapes × stream folds). With thousands of shapes, per-shape DS reads at boot are too slow. Boot does: load catalog index → register predicates into the shared routers (cheap, in-memory) → resume per-table tailers from the min persisted offset per table (one shared replay per table catches every shape up — no per-shape work). Expensive per-shape state is rebuilt on first touch: /v1/shape key sets by folding the shape stream on first unknown-handle request (no more restart 409 stampede), gates only if a backfill was incomplete.
• Incomplete backfills are discarded at boot (catalog row + stream removed), same policy as upstream (shape_status.ex:54-81).
• Collision-free shape ids across restarts (persist the counter or random ids) — fixes Shape id reuse after engine restart can append onto stale shape/<id> streams #3.
• Orphan GC: background sweep of DS shape/* streams not present in the catalog.

Interactions: table-stream trimming (future) must never trim below the min persisted tailer offset; dormant-shape reactivation (retention issue, TBD) reuses the same lazy catch-up path.

🤖 Generated with Claude Code