diff --git a/deploy/helm/pacer/templates/NOTES.txt b/deploy/helm/pacer/templates/NOTES.txt index ffda963..258ab71 100644 --- a/deploy/helm/pacer/templates/NOTES.txt +++ b/deploy/helm/pacer/templates/NOTES.txt @@ -1,4 +1,11 @@ PACER {{ .Chart.AppVersion }} installed as {{ .Release.Name }} in namespace {{ .Release.Namespace }}. +{{- if include "pacer.scatterEnabled" . }} + +Write scatter is ON (ADR-0032): a PUT of a new key becomes a multipart upload the +coordinating node creates directly on the bucket. If that node dies before it can abort +one, the client never saw an upload ID to clean it up with. Give the bucket an +AbortIncompleteMultipartUpload lifecycle rule — see docs/helm/write-scatter.md. +{{- end }} {{- if include "pacer.requesterMode" . }} auth.mode=requester (ADR-0041): S3 authorizes each caller's OWN signature. Configure the diff --git a/deploy/helm/pacer/tests/write_scatter_test.yaml b/deploy/helm/pacer/tests/write_scatter_test.yaml new file mode 100644 index 0000000..c89840f --- /dev/null +++ b/deploy/helm/pacer/tests/write_scatter_test.yaml @@ -0,0 +1,27 @@ +# NOTES.txt's write-scatter lifecycle-rule notice (ADR-0032): a coordinator that dies +# mid-upload leaves a multipart upload the client never saw an ID for, so the operator's +# only way to learn the bucket needs an AbortIncompleteMultipartUpload rule is to be told — +# nothing fails loudly without it, the cost just accrues silently. +suite: write scatter NOTES.txt notice (ADR-0032) +templates: + - NOTES.txt +tests: + - it: "a Standard backend's default-on scatter prints the lifecycle-rule notice" + set: {config.backendType: standard} + asserts: + - matchRegexRaw: + pattern: "AbortIncompleteMultipartUpload" + template: NOTES.txt + + - it: "Express leaves the scatter off by default and prints no notice" + asserts: + - notMatchRegexRaw: + pattern: "AbortIncompleteMultipartUpload" + template: NOTES.txt + + - it: "an explicit scatter.enabled=false on Standard silences the notice" + set: {config.backendType: standard, scatter.enabled: false} + asserts: + - notMatchRegexRaw: + pattern: "AbortIncompleteMultipartUpload" + template: NOTES.txt diff --git a/docs/helm/write-scatter.md b/docs/helm/write-scatter.md index ad48686..43c6137 100644 --- a/docs/helm/write-scatter.md +++ b/docs/helm/write-scatter.md @@ -372,6 +372,50 @@ which no template can know. Rule 3 of the sizing block above `resources:` in `values.yaml` is where the operator sizes it, and `pacer_cgroup_memory_file_bytes` is where they watch it. +## The bucket needs an `AbortIncompleteMultipartUpload` lifecycle rule + +A write scatter creates its own multipart upload directly on the real bucket (ADR-0032 +§ 2), and the coordinating node — not the client — holds the only upload ID. If that +coordinator dies after `CreateMultipartUpload` and before `coordinate::unwind` +(`crates/pacer-daemon/src/coordinate.rs`) gets to call `AbortMultipartUpload`, the upload +is left incomplete: the client never saw an upload ID to abort it with, so nothing on the +client side can clean it up, and S3 keeps billing the staged parts until something does. +`unwind`'s own doc comment calls this out as exactly what the lifecycle rule is for — it +only covers a coordinator that survives long enough to reach it. + +Give the bucket the rule regardless of whether the scatter is on for it. PACER also +proxies a client's own multipart upload unchanged on every backend (ADR-0032 +Consequences, "a client-supplied multipart upload"), and that upload is just as +orphanable if the client dies mid-upload — the rule is general bucket hygiene for +anything written through PACER, and load-bearing specifically for the scatter's own +uploads, where the client-side abort path does not exist at all. + +General-purpose (Standard) buckets: + +```bash +aws s3api put-bucket-lifecycle-configuration --bucket \ + --lifecycle-configuration '{"Rules":[ + {"ID":"abort-incomplete-multipart-uploads","Status":"Enabled","Filter":{"Prefix":""}, + "AbortIncompleteMultipartUpload":{"DaysAfterInitiation":7}}]}' +``` + +S3 Express One Zone directory buckets support lifecycle rules too, restricted to +expiration actions (ADR-0002's "expiration-only lifecycle" limitation); +`AbortIncompleteMultipartUpload` is one of them, so the same rule applies, addressed by +the directory bucket's own name: + +```bash +aws s3api put-bucket-lifecycle-configuration --bucket --use2-az1--x-s3 \ + --lifecycle-configuration '{"Rules":[ + {"ID":"abort-incomplete-multipart-uploads","Status":"Enabled","Filter":{"Prefix":""}, + "AbortIncompleteMultipartUpload":{"DaysAfterInitiation":7}}]}' +``` + +7 days is a starting point, not a measurement: long enough that it never races a write +that is still legitimately in flight, short enough that a crashed coordinator is not +billed for staged parts indefinitely. Tighten it once you know how long your slowest +write actually takes. + ## See also * [memory-model.md](memory-model.md) — the memory sizing rule, including the diff --git a/docs/install.md b/docs/install.md index 75908ed..e76ae32 100644 --- a/docs/install.md +++ b/docs/install.md @@ -8,7 +8,12 @@ - **A bucket.** The default is an S3 Express One Zone directory bucket in the same AZ (by AZ ID) as the cache nodes, with a `com.amazonaws..s3express` Gateway VPC endpoint. PrivateLink is not supported for Express. For S3 Standard, set - `config.backendType: standard`. + `config.backendType: standard`. Whichever backend, give the bucket an + `AbortIncompleteMultipartUpload` lifecycle rule before writing through PACER: the write + scatter (Standard only, on by default — see [features.md](features.md)) creates + multipart uploads the client never sees an ID for, so only this rule or a surviving + coordinator can clean one up after a crash. [write-scatter.md](helm/write-scatter.md) + has the rule's form for both backends. - **EKS Pod Identity** for the chart's ServiceAccount, with permission for the bucket's operations (`s3express:CreateSession` for Express). - **Cache nodes**, ideally in one AZ, with instance-store NVMe (RAID0), and Nitro v4 or later