Skip to content

perf(s3): coalesce the bulk-submit change log to one object per batch - #1434

Open
angela-helios wants to merge 1 commit into
fix/945-reland-s3-ingest-guard-and-rawfrom
perf/1429-coalesce-change-log
Open

angela-helios wants to merge 1 commit into
fix/945-reland-s3-ingest-guard-and-rawfrom
perf/1429-coalesce-change-log

Conversation

@angela-helios

Copy link
Copy Markdown
Contributor

Why

S3 $bulk-submit ingest wrote one change-log object per resource — one of the per-entry PUTs. With the raw archive already coalesced (#1411 / the re-land in #1433), that change-log PUT is the next-cheapest to drop: the change log has no per-object reader (rollback lists the whole changes/ prefix and sorts), so writing it once per batch instead of once per resource costs nothing on read. This takes the concurrent fast-load path from 6 → 5 PUTs/resource (#1429).

Stacked on #1433. Base is fix/945-reland-s3-ingest-guard-and-raw so the diff here is only the change-log change; GitHub retargets it to main when #1433 merges. Review/merge #1433 first.

What

  • process_single_entry now returns (BulkEntryResult, Option<SubmissionChange>) instead of calling record_change — the batch gathers every change (from both the concurrent and serial paths) and writes them in one coalesced object via persist_change_batch, keyed by first line and discriminated by file_url, exactly as the raw archive and the entry receipts are (submit_change_batch_key).
  • The coalesced write happens after the resources and receipts, before notify_batch_committed; its error is reported after the receipts and yields to the walk error (the ui: 1h "FHIR Resources over Time" chart stays on "Waiting for the live figures…" during a large import — dashboard blocks on storage scans instead of degrading to approximate data #1078 "report what was written" contract). Whatever changes a partially-failed batch produced are still written, so rollback coverage is preserved.
  • load_changes reads both shapes under changes/: the coalesced array and a legacy single-change object — the rollback trait's record_change is untouched and still writes the single form for composite backends, so mixed submissions roll back fully.

Test

s3::tests bulk-submit suite green (29): two new — a three-entry batch records its changes in one object and list_changes reads all three back; load_changes reads a submission holding both a coalesced batch object and a legacy single-change object. The existing duplicate_abort_and_rollback already exercises coalesced-write + dual-format-read + rollback end-to-end. fmt + clippy clean under the CI allow-flags.

Perf

hfs-50 to A/B on MinIO (objects/resource ~6.0 → ~5.0, throughput +~12–15%, per #1429) and confirm rollback + dual-format correctness before this is considered validated.

Ingest wrote one change-log object per resource — one of the per-entry
PUTs. The change log has no per-object reader (rollback lists the whole
`changes/` prefix), so a batch now returns each change up from
process_single_entry and writes them all in a single coalesced object,
keyed by first line and discriminated by file_url exactly as the raw
archive and the entry receipts are. That drops one PUT per resource (6->5
after the raw-archive coalescing).

load_changes reads both shapes under `changes/`: the coalesced array and
a legacy single-change object — the rollback trait's record_change still
writes the single form for composite backends. Whatever changes a
partially-failed batch produced are still written before its error
propagates, so rollback coverage is preserved.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant