perf(s3): coalesce the bulk-submit change log to one object per batch - #1434
Open
angela-helios wants to merge 1 commit into
Open
angela-helios wants to merge 1 commit into
angela-helios wants to merge 1 commit into
Conversation
Ingest wrote one change-log object per resource — one of the per-entry PUTs. The change log has no per-object reader (rollback lists the whole `changes/` prefix), so a batch now returns each change up from process_single_entry and writes them all in a single coalesced object, keyed by first line and discriminated by file_url exactly as the raw archive and the entry receipts are. That drops one PUT per resource (6->5 after the raw-archive coalescing). load_changes reads both shapes under `changes/`: the coalesced array and a legacy single-change object — the rollback trait's record_change still writes the single form for composite backends. Whatever changes a partially-failed batch produced are still written before its error propagates, so rollback coverage is preserved.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
S3
$bulk-submitingest wrote one change-log object per resource — one of the per-entry PUTs. With the raw archive already coalesced (#1411 / the re-land in #1433), that change-log PUT is the next-cheapest to drop: the change log has no per-object reader (rollback lists the wholechanges/prefix and sorts), so writing it once per batch instead of once per resource costs nothing on read. This takes the concurrent fast-load path from 6 → 5 PUTs/resource (#1429).What
process_single_entrynow returns(BulkEntryResult, Option<SubmissionChange>)instead of callingrecord_change— the batch gathers every change (from both the concurrent and serial paths) and writes them in one coalesced object viapersist_change_batch, keyed by first line and discriminated byfile_url, exactly as the raw archive and the entry receipts are (submit_change_batch_key).notify_batch_committed; its error is reported after the receipts and yields to the walk error (the ui: 1h "FHIR Resources over Time" chart stays on "Waiting for the live figures…" during a large import — dashboard blocks on storage scans instead of degrading to approximate data #1078 "report what was written" contract). Whatever changes a partially-failed batch produced are still written, so rollback coverage is preserved.load_changesreads both shapes underchanges/: the coalesced array and a legacy single-change object — the rollback trait'srecord_changeis untouched and still writes the single form for composite backends, so mixed submissions roll back fully.Test
s3::testsbulk-submit suite green (29): two new — a three-entry batch records its changes in one object andlist_changesreads all three back;load_changesreads a submission holding both a coalesced batch object and a legacy single-change object. The existingduplicate_abort_and_rollbackalready exercises coalesced-write + dual-format-read + rollback end-to-end. fmt + clippy clean under the CI allow-flags.Perf
hfs-50 to A/B on MinIO (objects/resource ~6.0 → ~5.0, throughput +~12–15%, per #1429) and confirm rollback + dual-format correctness before this is considered validated.