Skip to content

fix(purchase): user-facing Retry ignores recIsSafeToRedrive, so retrying a landed Azure savings-plan buys a second one #1668

Description

@cristim

Surfaced during the adversarial review of #1655 (fix for #1537). Pre-existing on main; #1655 does not cause it and does not fix it.

What

purchase.recIsSafeToRedrive (internal/purchase/manager.go:290-307) already encodes the fact that an Azure savings-plans purchase is not safe to re-drive:

case "azure":
    // Azure savings-plans uses a timestamp-based alias name and has no
    // server-side idempotency key, so a re-drive would create a duplicate.
    // All other Azure services use DoIdempotentPurchaseTwoStep (#729).
    return rec.Service != "savingsplans" && rec.Service != "savings-plans"

Its wrapper allRecsSafeToRedrive is called from exactly two places, both in the reaper:

  • internal/purchase/reaper.go:224 — deciding whether to append "; safe to retry" to the failure note
  • internal/purchase/manager.go:469 — RecoverStrandedApprovals, gating an in-place re-drive

It is never called from the user-facing retry path. loadAndValidateRetryRequest / checkRetryRateGates / persistRetryExecution (internal/api/handler_purchases.go) gate on status, RBAC, already-retried, resolveOpsHint, and the retry-attempt threshold — nothing consults provider re-drive safety.

Why it costs money

Every other provider path reproduces a deterministic provider token from the execution's lineage key, so a retry of an execution whose commitment actually landed short-circuits at the provider:

provider / service dedupe mechanism
AWS, all services ClientToken (Savings Plans) or the EC2 RI tag-guard
Azure reservations DoIdempotentPurchaseTwoStep lookup (#729)
GCP compute CUDs RequestId + deterministic name derived from the token (#654)
Azure savings-plans none — order alias named from time.Now().UnixNano()

So for an Azure SP row that failed after the order landed (a timeout, a post-purchase history write failure, a lost response), the operator sees a failed row with a Retry button, clicks it, and gets a second savings plan. Unlike every other path there is no server-side guard behind it, and a savings plan cannot be cancelled — it is a multi-year commitment.

Nothing warns the operator either: the History UI hides Retry only when resolveOpsHint matches a known-persistent failure string, and none of those strings relate to provider re-drive safety.

Suggested fix

Consult the existing helper on the user-facing path, rather than adding a second notion of re-drive safety:

  • In loadAndValidateRetryRequest (or checkRetryRateGates, next to the ops-hint gate), reject with a 409 + a specific reason when any selected rec fails recIsSafeToRedrive — the same shape as the existing ops_hint 409, so the frontend already renders it in place of the Retry button.
  • Decide explicitly whether ?force=true may override it. Given a savings plan cannot be cancelled, defaulting to not overridable seems right; if it is overridable it should require a distinct confirm, not the existing generic threshold confirm.
  • recIsSafeToRedrive's default: return false also means an unknown provider is currently retryable from the API while being un-redrivable by the reaper — worth aligning in the same change.

Regression test

Assert on purchase counts, not statuses: an azure/savingsplans failed row must be refused by the retry endpoint with zero successor rows created, while an azure/compute row with the same shape is still retryable. internal/api/handler_purchases_retry_fanout_test.go (added in #1655) has a harness that drives the real retry handler into a real purchase.Manager and counts commitments reaching the provider; the same shape fits here.

Related

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions