Repository navigation
fix(aws): EC2 and Redshift purchases are re-sent by the SDK after a lost response (double-buy) #296
Description
Summary
EC2 Reserved Instance and Redshift reserved node purchases can double-buy when AWS commits the purchase but the response is lost, because the SDK retries a non-idempotent call.
Current behaviour
providers/aws/internal/purchasecfg/config.go sets RetryMaxAttempts = 2 for every purchase client. ec2.PurchaseReservedInstancesOffering (providers/aws/services/ec2/client.go) and redshift.PurchaseReservedNodeOffering (providers/aws/services/redshift/client.go) have no ClientToken and no caller-chosen ID (verified on the aws-sdk-go-v2 input structs). If the response is lost after AWS commits (connection drop, 15s HTTP timeout, 5xx), the SDK retries and buys a second commitment. The idempotency-tag guard runs only before the first attempt and the tag is written after the buy, so the first RI is invisible. RDS, ElastiCache, MemoryDB and OpenSearch reuse one deterministic reservation ID per call (a retry gets AlreadyExists) and Savings Plans sends a ClientToken, so they are not affected.
Steps to verify the gap
Point the real SDK client built by NewClient at a local server that drops the connection after receiving Action=PurchaseReservedInstancesOffering (or PurchaseReservedNodeOffering) and count purchase requests. On origin/main it receives 2. No AWS calls, no purchases.
Expected behaviour
The purchase request is sent once. A lost response is reported as an outcome-unknown error so callers do not treat it as safely re-drivable.
Proposed fix
Per-operation RetryMaxAttempts = 1 on the two purchase calls and purchasecfg.ClassifyPurchaseError: only a definite service rejection (APIError with HTTP 4xx, or a throttle code) is returned unchanged; everything else wraps purchasecfg.ErrOutcomeUnknown. Platform follow-up: manager.go treats all AWS as redrive-safe and the reaper says "safe to retry"; EC2/Redshift outcome-unknown must be refused there, plus the pin bump.
References and severity
MON-02 in the 2026-10-08 fleet review (rv-platform-money H2). High severity when triggered: a duplicate non-cancelable 1y/3y commitment. Related: go#53, platform#243, #667, #683.
Activity
cristim commentedon Oct 9, 2026
Go half merged: PR #297 squash-merged as bcc0dfc (head 082df24, full CI green after a rerun of a golangci-lint schema-fetch flake; independent review by deployment-investigation-kimi, final ack at 082df24).
What changed: EC2 and Redshift purchases are sent exactly once (SDK retries disabled per operation) and a lost response returns an error wrapping
purchasecfg.ErrOutcomeUnknown. Tested against a server that drops the connection: 1 request, outcome-unknown error.Still open, so this issue stays open: the platform must stop treating AWS EC2/Redshift purchases as redrive-safe (manager.go, reaper, Retry gate), reconcile against what AWS bought before any re-buy, close the untagged-commit gap, and bump the pin to bcc0dfc. #296 closes when that platform PR merges.
docs(purchase): correct comments on EC2 Tenancy and Scope defaults
docs(purchase): state exactly what the EC2 client rejects before sending
fix(purchase): refuse retry and re-drive of AWS EC2 and Redshift purc…
cristim commentedon Oct 9, 2026
Platform side verified and merged: LeanerCloud/cloud-commitments-platform#778, squash merge commit 525e6ad40a7d0d14b8727840632ec6b45855e422 (gated at head bc5efe3f).
Verification at bc5efe3f (git archive, go1.26.9, macOS):
go build ./...,go test ./internal/purchase/... ./internal/api/...andgo test -race -tags=integration ./internal/purchase/...all exit 0. The refusal tests pass (TestAWSNoTokenRetrySuccessorIsRefusedAndFirstBuyStillExecutes, TestAWSRedriveRefusalSetIsExact, TestAWSRedriveGuardReachMatchesDispatchReach, TestRecoverStrandedApprovals_EC2RowIsFailedNotRedriven, TestRetryOfFailedAWSEC2OrRedshiftRowIsRefused). They also pass alongside #776's history dedupe and empty-CommitmentID tests. CI was 25/25 green. Reviews: Opus and kimi APPROVE, with an Opus delta review of the later comment commits. Mock and Postgres-container evidence only; no live AWS calls.Behaviour cost: the platform refuses Retry and re-drive of AWS EC2 and Redshift purchases after any failure, including definite rejections, because the provider's ErrOutcomeUnknown sentinel is internal.
?forcedoes not override the refusal. First purchases still execute.Follow-ups: #298, #301 (export ErrOutcomeUnknown so definite failures can be retried), LeanerCloud/cloud-commitments-platform#780.
Recommend closing this issue.