Repository navigation
fix(mem_wal): stream conditional file promotion - #9704
Closed
jackye1995 wants to merge 6 commits into
Closed
jackye1995 wants to merge 6 commits into
jackye1995 wants to merge 6 commits into
Conversation
Route fixed-file promotion through client-streamed create-if-absent uploads so large S3 flushes avoid server-side copy timeouts without weakening writer fencing.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
github-actions
Bot
added
A-python
Clean up conditional multipart uploads when destination creation is rejected.
Apply the same equal-size idempotency rule to conditional local copies as cloud multipart promotion.
![lance-gatekeeper[bot]](https://avatars.githubusercontent.com/in/4433141?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
lance-gatekeeper
Bot
added
the
K-changes
lance-gatekeeper
Bot
removed
the
K-changes
Contributor
There was a problem hiding this comment.
✅ Gate recommendation: approve.
The backend fallback and wrapper fix address both reported regressions. Native S3 can use conditional streaming while unverified stores retain their established conditional-copy path. No remaining blockers found.
lance-gatekeeper
Bot
added
the
K-approved
hamersaw
self-requested a review
October 8, 2026 14:33
Contributor
Author
|
Superseded by #9717, which now contains this complete streamed-promotion series together with the Blob v2 grouping and WAL-trigger optimizations validated as one stack. |
jackye1995
added a commit
that referenced
this pull request
Oct 13, 2026
Combines the complete MemWAL optimization stack and supersedes #9704. - Shares one generation-scoped Blob v2 pack across concurrent puts, fences failed groups, and preserves replay ordering. - Coalesces best-effort background WAL flush triggers while keeping freeze and close completion boundaries exact. - Preserves the 100 ms sequential flush cadence: elapsed intervals may prompt an append without fixing its batch boundary. - Replaces fixed-file S3 server-side copy with backend-gated, client-streamed conditional promotion, including multipart cleanup and idempotent retries. ## Benchmark protocol | Setting | Value | | --- | --- | | Benchmarked production revision | `9c0dcd606d02218f1ed8a54aaa46f71c41f7817e` (the follow-up commit only renames regression tests) | | Repetitions | One discarded full prewarm, then three measured repetitions per arm | | Workloads | LAION-100K multimodal KV; deterministic one-million-row non-blob integer and UUID KV | | Durability | 100 ms WAL interval; 64 MiB SSTable threshold/target; flushed before completion rate | | S3 provider-native server-side copy | Disabled in every reported S3 Lance arm | ## LAION-100K multimodal KV on S3 | Window | Lance rows/s | SlateDB rows/s | Lance / SlateDB (95% bootstrap) | Objects, L / S | Persisted bytes, L / S | | ---: | ---: | ---: | ---: | ---: | ---: | | 1 | 572 | 716 | 0.794 (0.789--0.826) | 921 / 582 | 7.280 / 14.059 GB | | 32 | 1,270 | 1,142 | 1.112 (1.001--1.190) | 463 / 154 | 7.279 / 14.058 GB | The exact approved head preserves the target object-store result: Lance is 1.11x faster on S3 at window 32 while persisting 48.2% fewer bytes. Against the prior valid campaign, window-32 throughput changed by -2.9% and its object count moved from 459 to 463, both within run-to-run variation. SlateDB remains 1.26x faster for sequential S3 writes. ## One-million-row non-blob control on S3 | Key | Window | Lance Krows/s | SlateDB Krows/s | Lance / SlateDB (95% bootstrap) | Objects, L / S | Lance byte saving | Point-read p50 | | --- | ---: | ---: | ---: | ---: | ---: | ---: | --- | | Integer | 1 | 11.60 | 9.57 | 1.264 (1.202--1.344) | 1,011 / 1,013 | 44.7% | Lance 2.84x lower | | Integer | 32 | 147.10 | 129.85 | 1.133 (1.095--1.214) | 74 / 62 | 44.9% | Lance 3.13x lower | | UUID | 1 | 13.09 | 9.49 | 1.381 (1.380--1.401) | 1,011 / 1,013 | 26.7% | Lance 5.75x lower | | UUID | 32 | 132.28 | 139.94 | 0.946 (0.930--0.963) | 74 / 65 | 26.8% | Lance 5.17x lower | The cadence fix materially improves sequential Lance throughput and keeps integer/window-32 ahead. UUID/window-32 is 5.4% below SlateDB, while Lance retains lower bytes and substantially lower S3 point-read latency. The earlier Lance UUID/window-32 runs ranged from 111 to 152 Krows/s; the new 130--138 Krows/s range differs by about 3% in mean and does not establish a throughput regression. The corrected cadence increases non-blob window-32 WAL objects from the earlier 44--46 to 74 because elapsed intervals can no longer be postponed by later writes; the previous claim that Lance always produced no more objects than SlateDB has therefore been removed. ## Verification - Lance Gatekeeper: `approve with a non-blocking risk` on the current test-only head; aggregate streaming-promotion memory scales with concurrently active shard writers. - CI: all branch-relevant checks pass, including the clean Windows rerun, all-feature Rust Clippy, and formatting. Compatibility Tests has the same `pylance==14.0.0b10`/`lance-namespace` dependency-resolution failure as current `main`. - Local: `cargo test -p lance --lib dataset::mem_wal` passes 789 tests with one ignored; all three shared-blob-pack durability regressions pass after the test-name cleanup; exact all-feature Clippy passes. - SELF review round 6: clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Validation
A full 1,161,228-row LAION-1M Blob v2 MemWAL campaign completed on S3 with one discarded prewarm and one measured trial. The measured run completed at 626.9 rows/s, passed all 101 key and image checksum samples, and completed the formerly failing 3.49-GiB conditional SSTable promotion without
CopyObjectorUploadPartCopy.