feat(purchases): flip RecoverStrandedApprovals to idempotent re-drive (now that #636 makes it safe)

[cristim]

Final step of the stranded-approval story. Depends on both PR #635 (recovery sweep that currently safe-FAILS stranded approved rows) and PR #638 (#636 idempotent commitment creation) merging.

Once both are in: change RecoverStrandedApprovals to re-drive a stranded approved execution instead of failing it, since a re-drive is now double-buy-safe:

• Savings Plans: native ClientToken (deterministic per feat(purchases): idempotent commitment creation (ClientToken/dedupe) (closes #636) #638) dedupes server-side.
• EC2 RI: tag-guard (tag:cudly-idempotency-token + active/payment-pending lookup) short-circuits a repeat, fails-loud on lookup error. Residual succeed-then-tag-fail window is irreducible and acceptable for an automated re-drive (the operator still sees the row).

Add a regression test: a stranded row is re-driven, and a re-drive that hits an already-existing commitment does NOT create a second one.

Note: #636's body and #632's analysis incorrectly assumed SP has no native idempotency; #638 corrected that. This issue reflects the corrected, safer reality.

[cristim]

Blocked by #641. The purchase-workflow trace found that idempotency (from #636) is honored ONLY by EC2 RI + Savings Plans. Every other executor — AWS RDS/ElastiCache/MemoryDB/OpenSearch/Redshift, all Azure reservations, and GCP — silently ignores opts.IdempotencyToken. Flipping RecoverStrandedApprovals to auto-re-drive (this issue) before #641 closes would double-purchase on every non-EC2/SP commitment. Do not implement #639 until #641 lands.

[cristim]

Blocker: Azure two-step reservation purchases are not idempotent — re-drive will double-buy

Implementation paused. Investigation surfaced an unresolved gap in the #641 prerequisite chain that makes auto-re-drive unsafe today for all 7 Azure reservation services.

What is OK

Spot-checked one executor per "honoring opts.IdempotencyToken" claim — these are all good:

• AWS SP (providers/aws/services/savingsplans/client.go:205-206): sets CreateSavingsPlanInput.ClientToken = opts.IdempotencyToken — server-side dedup.
• AWS EC2 RI (providers/aws/services/ec2/client.go:126-137): findRIByIdempotencyToken lookup via tag:cudly-idempotency-token, short-circuits on hit, tags fresh purchases with the token.
• AWS RDS / MemoryDB / OpenSearch / Redshift / ElastiCache: IdempotentReservationID("<prefix>", opts.IdempotencyToken) (e.g. providers/aws/services/rds/client.go:203, providers/aws/services/memorydb/client.go:135) plus idempotencyGuard + recoverAlreadyExists — token-derived reservation ID, server rejects duplicate.
• GCP Compute (providers/gcp/services/computeengine/client.go:522-548): commitment name + RequestId both derived from opts.IdempotencyToken (regression test client_test.go:530-549 pins same-token-same-RequestId).

DeriveIdempotencyToken(execID, recIndex) itself is a pure sha256("<execID>:<recIndex>") (pkg/common/tokens.go:41-44) — fully reproducible across runs.

What is broken

All 7 Azure reservation executors lost client-side idempotency in PR #680 (commit 65ecbf81c, merged after #653/#641):

cache, compute, cosmosdb, database, search, managedredis, synapse — every one now delegates the purchase to providers/azure/services/internal/reservations/DoPurchaseTwoStep (purchase.go:106-133). That helper:

1. POSTs calculatePrice — Azure mints a fresh reservationOrderId server-side every call (doCalculatePrice, purchase.go:137-164).
2. POSTs reservationOrders/{azure-id}/purchase.

opts.IdempotencyToken is not threaded into either step. The reservation order ID that #653 derived from the token via IdempotencyGUID is no longer used by these services. The PR #680 commit body explicitly acknowledges this and promises:

"Idempotency (Option B): Azure mints the order ID at calculatePrice time so client-supplied IDs are no longer feasible. Caller-level deduplication uses the purchase-automation tag already attached to each reservation request."

The caller-level guard does not exist. internal/purchase/execution.go:710 executeSinglePurchase calls serviceClient.PurchaseCommitment(recCtx, recommendation, opts) directly — no pre-flight check for an already-existing reservation tagged with the (executionID, recIndex) identity. The only tag attached is cudly-purchase-source: web|cli (PurchaseTagKey, e.g. providers/azure/services/compute/client.go:395), which is not unique per execution and so cannot deduplicate a re-drive.

A stranded Azure approval re-driven by RecoverStrandedApprovals will create a second reservation for every rec whose original attempt landed at Azure but didn't finalize the row. The compute client's own header comment (compute/client.go:409-412) describes the missing caller-level guard as if it already exists, which it doesn't.

Required to unblock #639

Either:

1. Restore client-side idempotency in the two-step flow — most surgical option is to derive a stable reservation order ID from opts.IdempotencyToken and PUT directly to reservationOrders/{derived-id} once calculatePrice validates the body, skipping Azure's server-minted ID. Requires Azure-side validation that PUT on a pre-known ID still works on the SKU families that motivated fix(providers/azure): direct PUT to reservationOrders/{id} rejected by Azure ("Session timed out - Call CalculatePrice again") — switch to two-step CalculatePrice→Purchase flow (P0) #677/fix(azure): switch 7 reservation clients to two-step calculatePrice->purchase flow (closes #677) #680.
2. Implement the promised caller-level guard — before executeSinglePurchase calls into an Azure service, list existing reservations tagged with cudly-idempotency-token: <token> (i.e. extend Azure executors to attach a per-rec identity tag, mirroring PurchaseTagKey) and short-circuit if found, mirroring findRIByIdempotencyToken on the EC2 RI side.

Either path needs its own issue + PR before #639 can land. AWS + GCP are ready; Azure is the lone hold-out.

Why this matters operationally

RecoverStrandedApprovals triggers on a 15-min approved-status threshold. Any Azure rec whose original execution hung past 15 min after the commitment landed (Lambda timeout, network partition between Azure and the row save, the partial-failure paths that #642 records) is a guaranteed double-buy under the proposed flip — which is exactly the scenario #639 is supposed to make safer.

Branch feat/639-recover-strands-auto-re-drive is parked (no code changes, only investigation) until the Azure path is closed.