From 02101df9a7d506b69bf0ff279bc65741fa7c9693 Mon Sep 17 00:00:00 2001 From: Alejandro Andraca Date: Fri, 2 Oct 2026 16:50:06 +0000 Subject: [PATCH 1/2] Document the AbortIncompleteMultipartUpload lifecycle rule for write scatter A write scatter creates its own multipart upload directly against the bucket, and the coordinating node holds the only upload ID. If that coordinator dies before it reaches its own unwind path, the client has no ID to abort the upload with, and the bucket keeps billing the orphaned parts until something else removes them. ADR-0032 and a code comment on coordinate::unwind already said the bucket needs an AbortIncompleteMultipartUpload lifecycle rule for exactly this case, but no user-facing doc did. Add that guidance to the install requirements and to the write-scatter design notes, with an example lifecycle configuration and a suggested expiry for both general-purpose (Standard) buckets and S3 Express One Zone directory buckets, and print a reminder in `helm install`'s NOTES.txt whenever the scatter resolves to enabled. A new chart test asserts the notice appears exactly when it should. --- deploy/helm/pacer/templates/NOTES.txt | 7 +++ .../helm/pacer/tests/write_scatter_test.yaml | 27 ++++++++++++ docs/helm/write-scatter.md | 44 +++++++++++++++++++ docs/install.md | 7 ++- 4 files changed, 84 insertions(+), 1 deletion(-) create mode 100644 deploy/helm/pacer/tests/write_scatter_test.yaml diff --git a/deploy/helm/pacer/templates/NOTES.txt b/deploy/helm/pacer/templates/NOTES.txt index ffda963..258ab71 100644 --- a/deploy/helm/pacer/templates/NOTES.txt +++ b/deploy/helm/pacer/templates/NOTES.txt @@ -1,4 +1,11 @@ PACER {{ .Chart.AppVersion }} installed as {{ .Release.Name }} in namespace {{ .Release.Namespace }}. +{{- if include "pacer.scatterEnabled" . }} + +Write scatter is ON (ADR-0032): a PUT of a new key becomes a multipart upload the +coordinating node creates directly on the bucket. If that node dies before it can abort +one, the client never saw an upload ID to clean it up with. Give the bucket an +AbortIncompleteMultipartUpload lifecycle rule — see docs/helm/write-scatter.md. +{{- end }} {{- if include "pacer.requesterMode" . }} auth.mode=requester (ADR-0041): S3 authorizes each caller's OWN signature. Configure the diff --git a/deploy/helm/pacer/tests/write_scatter_test.yaml b/deploy/helm/pacer/tests/write_scatter_test.yaml new file mode 100644 index 0000000..56839bd --- /dev/null +++ b/deploy/helm/pacer/tests/write_scatter_test.yaml @@ -0,0 +1,27 @@ +# NOTES.txt's write-scatter lifecycle-rule notice (ADR-0032): a coordinator that dies +# mid-upload leaves a multipart upload the client never saw an ID for, so the operator's +# only way to learn the bucket needs an AbortIncompleteMultipartUpload rule is to be told — +# nothing fails loudly without it, the cost just accrues silently. +suite: write scatter NOTES.txt notice (ADR-0032) +templates: + - NOTES.txt +tests: + - it: "a Standard backend's default-on scatter prints the lifecycle-rule notice" + set: { config.backendType: standard } + asserts: + - matchRegexRaw: + pattern: "AbortIncompleteMultipartUpload" + template: NOTES.txt + + - it: "Express leaves the scatter off by default and prints no notice" + asserts: + - notMatchRegexRaw: + pattern: "AbortIncompleteMultipartUpload" + template: NOTES.txt + + - it: "an explicit scatter.enabled=false on Standard silences the notice" + set: { config.backendType: standard, scatter.enabled: false } + asserts: + - notMatchRegexRaw: + pattern: "AbortIncompleteMultipartUpload" + template: NOTES.txt diff --git a/docs/helm/write-scatter.md b/docs/helm/write-scatter.md index ad48686..43c6137 100644 --- a/docs/helm/write-scatter.md +++ b/docs/helm/write-scatter.md @@ -372,6 +372,50 @@ which no template can know. Rule 3 of the sizing block above `resources:` in `values.yaml` is where the operator sizes it, and `pacer_cgroup_memory_file_bytes` is where they watch it. +## The bucket needs an `AbortIncompleteMultipartUpload` lifecycle rule + +A write scatter creates its own multipart upload directly on the real bucket (ADR-0032 +§ 2), and the coordinating node — not the client — holds the only upload ID. If that +coordinator dies after `CreateMultipartUpload` and before `coordinate::unwind` +(`crates/pacer-daemon/src/coordinate.rs`) gets to call `AbortMultipartUpload`, the upload +is left incomplete: the client never saw an upload ID to abort it with, so nothing on the +client side can clean it up, and S3 keeps billing the staged parts until something does. +`unwind`'s own doc comment calls this out as exactly what the lifecycle rule is for — it +only covers a coordinator that survives long enough to reach it. + +Give the bucket the rule regardless of whether the scatter is on for it. PACER also +proxies a client's own multipart upload unchanged on every backend (ADR-0032 +Consequences, "a client-supplied multipart upload"), and that upload is just as +orphanable if the client dies mid-upload — the rule is general bucket hygiene for +anything written through PACER, and load-bearing specifically for the scatter's own +uploads, where the client-side abort path does not exist at all. + +General-purpose (Standard) buckets: + +```bash +aws s3api put-bucket-lifecycle-configuration --bucket \ + --lifecycle-configuration '{"Rules":[ + {"ID":"abort-incomplete-multipart-uploads","Status":"Enabled","Filter":{"Prefix":""}, + "AbortIncompleteMultipartUpload":{"DaysAfterInitiation":7}}]}' +``` + +S3 Express One Zone directory buckets support lifecycle rules too, restricted to +expiration actions (ADR-0002's "expiration-only lifecycle" limitation); +`AbortIncompleteMultipartUpload` is one of them, so the same rule applies, addressed by +the directory bucket's own name: + +```bash +aws s3api put-bucket-lifecycle-configuration --bucket --use2-az1--x-s3 \ + --lifecycle-configuration '{"Rules":[ + {"ID":"abort-incomplete-multipart-uploads","Status":"Enabled","Filter":{"Prefix":""}, + "AbortIncompleteMultipartUpload":{"DaysAfterInitiation":7}}]}' +``` + +7 days is a starting point, not a measurement: long enough that it never races a write +that is still legitimately in flight, short enough that a crashed coordinator is not +billed for staged parts indefinitely. Tighten it once you know how long your slowest +write actually takes. + ## See also * [memory-model.md](memory-model.md) — the memory sizing rule, including the diff --git a/docs/install.md b/docs/install.md index 75908ed..e76ae32 100644 --- a/docs/install.md +++ b/docs/install.md @@ -8,7 +8,12 @@ - **A bucket.** The default is an S3 Express One Zone directory bucket in the same AZ (by AZ ID) as the cache nodes, with a `com.amazonaws..s3express` Gateway VPC endpoint. PrivateLink is not supported for Express. For S3 Standard, set - `config.backendType: standard`. + `config.backendType: standard`. Whichever backend, give the bucket an + `AbortIncompleteMultipartUpload` lifecycle rule before writing through PACER: the write + scatter (Standard only, on by default — see [features.md](features.md)) creates + multipart uploads the client never sees an ID for, so only this rule or a surviving + coordinator can clean one up after a crash. [write-scatter.md](helm/write-scatter.md) + has the rule's form for both backends. - **EKS Pod Identity** for the chart's ServiceAccount, with permission for the bucket's operations (`s3express:CreateSession` for Express). - **Cache nodes**, ideally in one AZ, with instance-store NVMe (RAID0), and Nitro v4 or later From 5154f968ca35c662b72300166ef93ef8b3ac5912 Mon Sep 17 00:00:00 2001 From: Alejandro Andraca Date: Mon, 5 Oct 2026 14:42:49 +0000 Subject: [PATCH 2/2] Fix yamllint brace spacing in the write-scatter NOTES.txt test --- deploy/helm/pacer/tests/write_scatter_test.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/deploy/helm/pacer/tests/write_scatter_test.yaml b/deploy/helm/pacer/tests/write_scatter_test.yaml index 56839bd..c89840f 100644 --- a/deploy/helm/pacer/tests/write_scatter_test.yaml +++ b/deploy/helm/pacer/tests/write_scatter_test.yaml @@ -7,7 +7,7 @@ templates: - NOTES.txt tests: - it: "a Standard backend's default-on scatter prints the lifecycle-rule notice" - set: { config.backendType: standard } + set: {config.backendType: standard} asserts: - matchRegexRaw: pattern: "AbortIncompleteMultipartUpload" @@ -20,7 +20,7 @@ tests: template: NOTES.txt - it: "an explicit scatter.enabled=false on Standard silences the notice" - set: { config.backendType: standard, scatter.enabled: false } + set: {config.backendType: standard, scatter.enabled: false} asserts: - notMatchRegexRaw: pattern: "AbortIncompleteMultipartUpload"