perf(mem_wal): optimize Blob v2 writes and promotion

[jackye1995]

Combines the complete MemWAL optimization stack and supersedes #9704.

• Shares one generation-scoped Blob v2 pack across concurrent puts, fences failed groups, and preserves replay ordering.
• Coalesces best-effort background WAL flush triggers while keeping freeze and close completion boundaries exact.
• Preserves the 100 ms sequential flush cadence: elapsed intervals may prompt an append without fixing its batch boundary.
• Replaces fixed-file S3 server-side copy with backend-gated, client-streamed conditional promotion, including multipart cleanup and idempotent retries.

Benchmark protocol

Setting	Value
Benchmarked production revision	9c0dcd606d02218f1ed8a54aaa46f71c41f7817e (the follow-up commit only renames regression tests)
Repetitions	One discarded full prewarm, then three measured repetitions per arm
Workloads	LAION-100K multimodal KV; deterministic one-million-row non-blob integer and UUID KV
Durability	100 ms WAL interval; 64 MiB SSTable threshold/target; flushed before completion rate
S3 provider-native server-side copy	Disabled in every reported S3 Lance arm

LAION-100K multimodal KV on S3

Window	Lance rows/s	SlateDB rows/s	Lance / SlateDB (95% bootstrap)	Objects, L / S	Persisted bytes, L / S
1	572	716	0.794 (0.789--0.826)	921 / 582	7.280 / 14.059 GB
32	1,270	1,142	1.112 (1.001--1.190)	463 / 154	7.279 / 14.058 GB

The exact approved head preserves the target object-store result: Lance is 1.11x faster on S3 at window 32 while persisting 48.2% fewer bytes. Against the prior valid campaign, window-32 throughput changed by -2.9% and its object count moved from 459 to 463, both within run-to-run variation. SlateDB remains 1.26x faster for sequential S3 writes.

One-million-row non-blob control on S3

Key	Window	Lance Krows/s	SlateDB Krows/s	Lance / SlateDB (95% bootstrap)	Objects, L / S	Lance byte saving	Point-read p50
Integer	1	11.60	9.57	1.264 (1.202--1.344)	1,011 / 1,013	44.7%	Lance 2.84x lower
Integer	32	147.10	129.85	1.133 (1.095--1.214)	74 / 62	44.9%	Lance 3.13x lower
UUID	1	13.09	9.49	1.381 (1.380--1.401)	1,011 / 1,013	26.7%	Lance 5.75x lower
UUID	32	132.28	139.94	0.946 (0.930--0.963)	74 / 65	26.8%	Lance 5.17x lower

The cadence fix materially improves sequential Lance throughput and keeps integer/window-32 ahead. UUID/window-32 is 5.4% below SlateDB, while Lance retains lower bytes and substantially lower S3 point-read latency. The earlier Lance UUID/window-32 runs ranged from 111 to 152 Krows/s; the new 130--138 Krows/s range differs by about 3% in mean and does not establish a throughput regression. The corrected cadence increases non-blob window-32 WAL objects from the earlier 44--46 to 74 because elapsed intervals can no longer be postponed by later writes; the previous claim that Lance always produced no more objects than SlateDB has therefore been removed.

Verification

• Lance Gatekeeper: approve with a non-blocking risk on the current test-only head; aggregate streaming-promotion memory scales with concurrently active shard writers.
• CI: all branch-relevant checks pass, including the clean Windows rerun, all-feature Rust Clippy, and formatting. Compatibility Tests has the same pylance==14.0.0b10/lance-namespace dependency-resolution failure as current main.
• Local: cargo test -p lance --lib dataset::mem_wal passes 789 tests with one ignored; all three shared-blob-pack durability regressions pass after the test-name cleanup; exact all-feature Clippy passes.
• SELF review round 6: clean.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[xuanyu-z]

Nice! Overall looks great!

[lance-gatekeeper]

⚠️ Gate recommendation: approve with a non-blocking risk.

Group packing, dynamic flush boundaries, and the cancellation fence remain sound. The accepted promotion trade-off adds read and upload buffers for each active shard writer. Aggregate memory grows with shard concurrency and multipart part sizes; include those buffers in deployment memory budgets while retaining per-shard flush serialization.