perf(rollout): reuse a resident MemWAL ShardWriter across appends

[beinan]

Summary

RolloutStore::add opened a brand-new ShardWriter for every append and immediately close()d it. Each open cold-rebuilds the object store — ObjectStore::from_uri constructs a fresh ObjectStoreRegistry::default(), i.e. a new connection pool — so every append paid a cold DNS resolution + TCP/TLS handshake plus a redundant claim_epoch, then threw the connection away.

On Azure blob (where DNS resolution is the slow part) this dominated append latency (p50 ~1.5s, tail 12–100s) and left pods with a pile of TIME_WAIT sockets and zero long-lived connections.

This holds one resident ShardWriter per shard and reuses it across appends, so the epoch is claimed once and the object-store connection pool is reused (DNS resolved once). Per append we now do put → force_seal_active → wait_for_flush_drain, which still commits a flushed generation to the shard manifest before returning — cross-instance read-after-write visibility is unchanged. Only the per-append open/claim/reconnect churn is removed.

Note: because we keep strong cross-instance visibility, the generation-per-visible-append (and the existing count/time-triggered self-merge that collapses them) is intentionally unchanged. This PR reduces latency, not generation count.

Details

• Resident writer: new write_writer: Option<ShardWriter> field, opened lazily via ensure_write_writer, reused across add.
• Epoch fencing: merge_own_shard calls claim_epoch, which fences a live writer, so it now close()s and clears the resident writer first; the next add transparently reopens against the fresh epoch.
• Fence self-heal: add retries once (reopening the writer) if an op returns a fence error.
• Lifecycle: ShardWriter has no Drop, so added pub async fn close() for graceful draining plus a best-effort Drop impl that spawns a detached close() when a Tokio runtime exists — avoids background-task leaks on LRU eviction.

Test plan

• cargo test -p lance-context-core (25 passed; includes 3 new tests)
  • add_reuses_resident_writer_epoch_stable — epoch does not advance per append; every append immediately visible via get_by_id.
  • merge_reopens_writer_after_epoch_claim — merge-per-append never surfaces a fence error; rows stay visible.
  • close_drains_resident_writer_and_add_reopens — close() drops the writer (idempotent); add reopens.
• cargo build -p lance-context-server (unchanged callers compile)
• cargo clippy -p lance-context-core --tests (clean)
• cargo fmt --all
• Manual e2e on Azure: watch p50 append latency drop and ESTABLISHED connections persist instead of TIME_WAIT churn.

🤖 Generated with Claude Code