Repository navigation
fix(azure): surface the read error that decides purchase retry-vs-abort - #1767
Conversation
📝 WalkthroughWalkthroughThe Azure reservation purchase path now preserves response-body read errors, classifies retryability as unknown for truncated 400 responses, and retries unreadable responses. Tests cover error propagation, retry limits, status handling, and non-retryable statuses. ChangesAzure purchase retryability
Estimated code review effort: 3 (Moderate) | ~20 minutes Possibly related issues
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
Independent review at
|
| scenario | origin/main (pre-fix) |
182f242a9 (post-fix) |
|---|---|---|
| 400, body cut short before the fragment | purchaseAttempts=1, IsSessionTimeout=false |
purchaseAttempts=1, IsSessionTimeout=false |
| 400, full body, fragment intact (control) | purchaseAttempts=3, IsSessionTimeout=true |
purchaseAttempts=3, IsSessionTimeout=true |
The control moves, so the probe is measuring the retry loop and not a constant. The defect row does not move.
Pre-fix error:
reservation purchase failed with status 400: {"error":{"c
Post-fix error:
reservation purchase failed with status 400 and its response body could not be read,
so retryability is unknown (partial body: "{\"error\":{\"c"): unexpected EOF reading response body
Same decision, richer text.
This is a genuine and worthwhile improvement — the issue has two halves ("the retry is abandoned" and "nothing in the message indicated the response had not been fully read"), and this closes the second one cleanly. But it is an observability fix, not a retry-classification fix, and the body and the code comment both describe it as the latter. The comment at purchase.go:484-490 ends "Surface the read failure rather than classifying retryability from a body that was never fully received" — the surfacing happens, the classification still does too.
Concretely: Closes LeanerCloud/cloud-commitments-cli#1766 overstates it. #1766's core sentence — "the purchase is abandoned — the exact condition the retry exists for" — is still true at this head.
Making the behaviour match the claim needs the unknown-retryability state carried out of band rather than in the string: a sentinel or typed error from doPurchase, and a branch in DoPurchaseTwoStep that treats errors.Is(err, errRetryabilityUnknown) as retryable alongside IsSessionTimeout. Worth noting that is safe here — a non-2xx means the purchase did not commit, and DoIdempotentPurchaseTwoStep already refuses to fall through on a lookup error — but it is a design call, not a one-liner, and it deserves its own change.
Either extend the PR to change the classification, or keep the scope and adjust the framing (retitle to the diagnostic it delivers, drop Closes to Refs, and leave #1766 open for the behavioural half). I would not close #1766 on this diff.
F2 (minor) — the "essential property" assertion is tautological post-fix
purchase_test.go:863 and :872:
assert.ErrorContains(t, err, "could not be read", ...) // :863
...
assert.True(t,
IsSessionTimeout(err) || strings.Contains(err.Error(), "could not be read"),
"an unreadable body must not silently produce a non-retryable classification") // :872Once :863 passes, the second disjunct of :872 is necessarily true. :872 cannot fail independently of :863 — it is asserting the same substring twice, the second time behind an || that can never be reached.
It is also the one assertion labelled as pinning retryability, and per F1 that property does not hold. A reader scanning the test comes away believing the retry classification is guarded. Given this repo just spent #1740/#1750 on assertions that read as guarantees they do not provide, this one is worth removing or replacing — the honest version drives DoPurchaseTwoStep and asserts the attempt count, which is exactly the assertion that would have caught F1.
Also minor: the body heads the section "Regression tests, both directions", but TestDoPurchase_UnreadableBodyOnSuccessStillSucceeds passes against pre-fix code (verified). It is a guard, not a regression test — correctly so, since that direction was never broken. Worth saying that rather than implying both fail pre-fix.
Verified clean
The success path is unchanged, and its guard genuinely bites. resp.StatusCode is checked before readErr, so an unreadable body cannot fail a completed purchase. Confirmed by mutation — hoisting the readErr check above the 2xx short-circuit:
--- FAIL: TestDoPurchase_UnreadableBodyOnSuccessStillSucceeds
MUTANT: read failed, status 200: unexpected EOF reading response body
This is the dangerous direction and it is properly nailed down.
The pre-fix failure re-derives verbatim. Checked out the PR's purchase_test.go onto origin/main production code:
purchase_test.go:863: Error "reservation purchase failed with status 400: {\"error\":{\"c" does not contain "could not be read"
purchase_test.go:865: Error "reservation purchase failed with status 400: {\"error\":{\"c" does not contain "unexpected EOF reading response body"
purchase_test.go:872: Should be true
--- FAIL: TestDoPurchase_UnreadableBodyDoesNotSilentlyLookPermanent
--- PASS: TestDoPurchase_UnreadableBodyOnSuccessStillSucceeds
Quoted text matches the body exactly.
The sibling assessment holds, and I widened it past the file. Swept every non-test discarded io.ReadAll error in providers/azure (5 sites; ERE with [[:space:]], sanity-checked against a line known to match):
| site | feeds | verdict |
|---|---|---|
purchase.go:287 doCalculatePrice |
message; truncation fails json.Unmarshal, empty ReservationOrderID separately guarded; caller returns immediately without classifying |
safe |
purchase.go:390 fetchReservationOrdersPage |
message; truncation fails json.Unmarshal; DoIdempotentPurchaseTwoStep:448 refuses to purchase on a lookup error |
safe, fails closed |
internal/pricing/retail_prices.go:110 |
message only, inside the non-200 branch | safe |
compute/client.go:406 triggerCapacityProviderRegistration |
message only | safe |
compute/client.go:372 fetchCapacityProviderState |
its result does feed control flow (state.RegistrationState == "Registered"), but a truncated 2xx fails json.Unmarshal → early return before the branch, and ensureCapacityProviderRegistered only logs a warning either way |
safe |
So :475 really was the only one where a discarded error reached a decision. That distinguishing property survives the change.
The golangci set diff reproduces exactly. 592 on main, 591 on head; the sole real delta is purchase.go:475:8: Error return value of io.ReadAll is not checked (errcheck), removed. A raw comm also surfaces 41 purchase_test.go entries on each side — every one is a +1 line shift from the added "strings" import, each NEW matching a REMOVED one line lower. Zero genuinely new findings; the new test code adds none of its own.
Gates at 182f242a9
| gate | result |
|---|---|
gofmt -l . |
clean (0 files) |
providers/azure: go build ./... + go vet ./... + go test -race -count=1 ./... |
exit 0 — 12 ok + 2 no-test-files = 14 packages, 0 FAIL |
gocyclo -over 10 -ignore "_test\.go" (root and providers/azure) |
exit 0, empty |
golangci-lint run @ CI-pinned v2.10.1, providers/azure |
exit 1 / 591 issues both sides bar the one removed — pre-existing debt per LeanerCloud/cloud-commitments-platform#174/#1759, neither added to nor masked |
Everything green. F1 is not a test or build failure — it is that the diff does less than it says, which no gate can see.
Verdict
Mergeable on its merits, not on its framing. The code is correct, narrowly scoped, and the risky direction is properly guarded. Before merge I would fix F2 and settle F1 one way or the other: either carry the unknown-retryability state out of band so the retry loop actually changes, or keep this scope and stop it closing #1766.
182f242 to
c4d2d85
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
providers/azure/services/internal/reservations/purchase.go (1)
494-509: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winSplit these files to meet the 500-line limit.
Both final files exceed 500 lines. Keep the extraction within the reservations bounded context.
providers/azure/services/internal/reservations/purchase.go#L494-L509: extract a cohesive private purchase-response classification unit.providers/azure/services/internal/reservations/purchase_test.go#L813-L948: move the truncation helpers and unreadable-body regression tests into a focused test file.As per coding guidelines, “keep files under 500 lines.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@providers/azure/services/internal/reservations/purchase.go` around lines 494 - 509, Split the reservations purchase code and tests so every file stays under 500 lines. In providers/azure/services/internal/reservations/purchase.go lines 494-509, extract the cohesive private purchase-response classification logic into a bounded-context helper while preserving status handling and unreadable-body retryability behavior. In providers/azure/services/internal/reservations/purchase_test.go lines 813-948, move the truncation helpers and unreadable-body regression tests into a focused reservations test file, updating references as needed without changing coverage.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@providers/azure/services/internal/reservations/purchase_test.go`:
- Around line 836-837: Update the comment near DoPurchaseTwoStep to describe the
retry condition as pre-fix behavior, using past tense and acknowledging that
retries were previously limited to IsSessionTimeout matches; do not alter the
purchase logic.
---
Nitpick comments:
In `@providers/azure/services/internal/reservations/purchase.go`:
- Around line 494-509: Split the reservations purchase code and tests so every
file stays under 500 lines. In
providers/azure/services/internal/reservations/purchase.go lines 494-509,
extract the cohesive private purchase-response classification logic into a
bounded-context helper while preserving status handling and unreadable-body
retryability behavior. In
providers/azure/services/internal/reservations/purchase_test.go lines 813-948,
move the truncation helpers and unreadable-body regression tests into a focused
reservations test file, updating references as needed without changing coverage.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 70d21cf1-9522-49de-8e20-20794b39ab83
📒 Files selected for processing (2)
providers/azure/services/internal/reservations/purchase.goproviders/azure/services/internal/reservations/purchase_test.go
doPurchase discarded the io.ReadAll error, and its error string is the only
input to IsSessionTimeout, which is the only thing DoPurchaseTwoStep consults
to decide whether to retry. A 400 whose body was truncated before the
"Session timed out" fragment therefore produced a bare `status 400: <partial>`
error, IsSessionTimeout returned false, and the purchase was abandoned on its
first attempt -- the exact condition purchaseMaxAttempts exists to survive --
with nothing in the message to say the response had not been fully read.
The failure direction is closed: it abandons a purchase rather than
double-buying. But a designed-retryable failure became permanent silently, and
the diagnostic that would explain it was the thing that went missing.
Retryability is no longer classified from a body that was never fully
received. On a non-2xx response with a failed read, the read error is wrapped
and surfaced along with the partial body. Success is still decided by the
status code alone, so an unreadable body cannot turn a completed purchase into
a reported failure.
Two regression tests, both driven by a body that yields 12 bytes and then
fails. The first pins that an unreadable body cannot silently produce a
non-retryable classification; it fails against the pre-fix code, where the
error read `reservation purchase failed with status 400: {"error":{"c` and
carried neither the read failure nor the session-timeout fragment. The second
guards the opposite direction, that a truncated body on a 200 still succeeds.
The two sibling io.ReadAll discards in this file were checked rather than
assumed and are not affected: :287 in doCalculatePrice and :390 in
fetchReservationOrdersPage both feed a json.Unmarshal that fails on
truncation, and :390's caller DoIdempotentPurchaseTwoStep explicitly refuses
to purchase on a lookup error rather than falling through.
golangci-lint on providers/azure, compared as a set diff rather than by
totals: zero new findings, one removed (the errcheck this fixes), 592 -> 591.
The totals alone would have hidden it -- the first attempt at this change
removed one finding and added two misspellings in new comments.
Closes #1766
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The previous commit surfaced the discarded read error in the message but did not change the retry decision. IsSessionTimeout still did strings.Contains on a string built from a truncated body, so a session-timeout 400 whose body did not read completely was still abandoned on its first attempt -- the exact condition purchaseMaxAttempts exists to survive. The message improved; the behaviour did not. Measured on the real loop with a counting client, minting a fresh response per attempt because the truncating body is stateful: readable session-timeout 400 3 purchase attempts (control) truncated session-timeout 400 1 purchase attempt (defect) The control moves, so the probe measures the loop rather than a constant. Retryability is now carried out of band. doPurchase wraps errRetryabilityUnknown when a non-2xx body fails to read, and DoPurchaseTwoStep treats errors.Is(err, errRetryabilityUnknown) as retryable alongside IsSessionTimeout. Retrying is safe here: a non-2xx means the order did not commit, the loop is bounded by purchaseMaxAttempts, and re-drives across invocations are guarded by DoIdempotentPurchaseTwoStep, which refuses to purchase when its idempotency lookup fails. The regression test is an ATTEMPT-COUNT assertion, not a message assertion. The previous test asserted on the error text and passed while the purchase was still abandoned, which is how the gap survived review; only counting attempts tells a changed message from changed behaviour. It fails against the message-only fix with expected 3, actual 1, and carries the readable control so it measures the loop rather than a constant. The "essential property" assertion it replaces was tautological once the message changed, while being labelled as pinning retryability -- a property that did not hold. That is the same defect class as #1595 and #1740: an assertion reading as a guarantee it does not provide. Drive-by, disclosed: a global spelling fix in this test file also corrected two pre-existing `behaviour` misspellings that main already carried. Set diff against main: zero new findings, three removed. Closes #1766 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
errRetryabilityUnknown fired on ANY non-2xx whose body failed to read -- 409, 500, 502, 504 alike -- which is far broader than the IsSessionTimeout path it sits beside, and broader than the defect it was written for. That breadth is load-bearing because the idempotency guard does not cover in-loop retries. DoIdempotentPurchaseTwoStep calls FindReservationOrderByIdempotencyToken exactly once, before DoPurchaseTwoStep is entered; the loop then runs up to purchaseMaxAttempts with no recheck between attempts, and each retry mints a fresh reservationOrderId via doCalculatePrice. Verified here: zero lookups occur inside the loop. So the guard catches a re-drive ACROSS invocations and nothing within one, which makes "a non-2xx means the order did not commit" the only thing standing between a truncated read and a double purchase -- and that premise is unverified for this API on 409 and 5xx. A truncated response is also mechanistically different from a cleanly read one: truncation is the transport-failure class associated with genuine commit ambiguity, since the server may have finished processing and begun writing when the connection died. A cleanly read 500 is a deliberate synchronous statement. The previous comment treated the two as equivalent. Issue #1766's defect is a 400 "Session timed out" with a truncated body, the only scenario in the issue, the PR description and both regression tests, so scoping to 400 closes it with no loss of coverage while removing the unverified commit-ambiguous statuses from the retry surface. Everything else keeps its pre-existing behavior. The sentinel's doc comment no longer asserts that a non-2xx means no commit. It states the scope, why the wider surface is excluded, and that the bound is purchaseMaxAttempts rather than the idempotency guard. Two tests pin the narrowing so it cannot be silently widened again: 409, 500, 502 and 504 with truncated bodies must not carry the sentinel, and a truncated 500 must be attempted exactly once. Both fail when the condition is widened back to any non-2xx. The attempt-count regression test still fails against the message-only revision with expected 3, actual 1. The missing in-loop idempotency recheck is pre-existing on the IsSessionTimeout path and is tracked as #1774; it is deliberately not addressed here. Refs #1766 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
c4d2d85 to
c3b629c
Compare
|
Merging. Closes #1766. Merging on independent adversarial review. The one CodeRabbit thread is a tense nitpick, answered in writing above rather than resolved, and it is now stale in a second way: the head moved after it was posted. The original defect: This PR took two rounds, and the first one did not work. The initial revision surfaced the read error in the message but left the decision unchanged. A reviewer drove the real retry loop with a counting client: The control moving is what proves the probe measures the loop rather than a constant. The author diagnosed its own error better than I did: it had asserted on the artifact it had just changed (the message) rather than the behaviour it claimed to fix (the retry). Counting attempts was the only thing that could distinguish them. The replacement test is an attempt-count assertion carrying the readable control, and it fails against the message-only revision with Then the safety question changed the shape of the fix. Asked whether any non-2xx can follow a purchase that actually committed, the reviewer could not resolve Azures behaviour from this repo and said so, but resolved the thing that decides it, by probe:
So the guard covers a re-drive across top-level invocations and nothing within one, and the bound is Hence the narrowing: Two tests pin the narrowing, because nothing else in the code would notice a re-widening: The success path is unchanged and its guard bites — Flagged by the author, unprompted: the loops trailing The in-loop idempotency gap is pre-existing on the Gates at |
Comment-only. Both make exhaustiveness claims that stopped being true when #1767 added a second retry trigger, and both now describe the current rule. purchase_test.go said doPurchase's error text is "the ONLY input to IsSessionTimeout, and DoPurchaseTwoStep retries only when that predicate matches". Neither half holds: the loop also retries on errors.Is(err, errRetryabilityUnknown). CodeRabbit flagged this on #1767; the PR merged before the fix landed, so the claim is live on main. DoPurchaseTwoStep's own doc had the same defect and nobody flagged it. It said retry happens on a "Session timed out" 400 and that "Other 4xx/5xx errors are returned immediately without retry", which omits the second trigger entirely. Found by sweeping the whole #1767 diff for the same class rather than fixing only the line that was reported. It now enumerates both cases and states that everything else -- including a cleanly read 4xx and any 5xx -- returns without retry. Rather than past-tensing the old text, each comment leads with the current rule and keeps the pre-fix explanation marked as the reason the test exists. A reader who stops after the first paragraph should come away with something true. Exhaustiveness claims are what readers rely on to decide they need not check, so a stale one is worse than none: #1757 (a live CSRF hole) survived behind a parity claim that was not true, and #1764's getAccountScope godoc described the inverse of its behavior after a change. Refs #1767, #1766 Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Closes #1766
The defect
doPurchasediscarded theio.ReadAllerror, and its error string is the only input to the retry decision:So a 400 whose body was truncated before the fragment produced a bare
status 400: <partial>,IsSessionTimeoutreturned false, and the purchase was abandoned on its first attempt — the exact conditionpurchaseMaxAttemptsexists to survive. Nothing in the message indicated the response had not been fully read.Direction: fails closed. It abandons a purchase, it does not double-buy — hence
severity/medium. But a designed-retryable failure became permanent silently, and the diagnostic that would have explained it is the thing that went missing.Found while triaging LeanerCloud/cloud-commitments-platform#174's 49 production findings. It is the only one of them where a discarded error feeds control flow rather than a message — full triage here.
The fix
Retryability is now carried out of band.
doPurchasewraps a newerrRetryabilityUnknownsentinel when a non-2xx body fails to read, andDoPurchaseTwoSteptreatserrors.Is(err, errRetryabilityUnknown)as retryable alongsideIsSessionTimeout. The retry decision no longer depends on text parsed from a body that never fully arrived, soCloses LeanerCloud/cloud-commitments-cli#1766now matches the diff.Scoped to 400 — matching
IsSessionTimeout's own status scope and the only scenario #1766 reports.That scoping is not cosmetic. The safety review established that the idempotency guard does not cover in-loop retries:
DoIdempotentPurchaseTwoStepcallsFindReservationOrderByIdempotencyTokenexactly once, beforeDoPurchaseTwoStepis entered, and there is no recheck between attempts — each retry mints a freshreservationOrderIdviadoCalculatePrice. Verified here: zero lookups occur inside the loop. So the guard covers a re-drive across invocations and nothing within one, and the bound ispurchaseMaxAttempts, not the guard.That makes "a non-2xx means the order did not commit" the only thing between a truncated read and a double purchase — and that premise is unverified for this API on 409 and 5xx. A truncated response is also mechanistically different from a cleanly read one: truncation is the transport-failure class associated with genuine commit ambiguity, since the server may have finished processing and begun writing when the connection died. A cleanly read 500 is a deliberate synchronous statement. An earlier revision of this PR treated the two as equivalent, and its safety comment asserted the non-commit premise outright; both are corrected.
Narrowing costs no coverage — #1766's defect is a 400 with a truncated body, the only case in the issue and in every regression test here — while removing the unverified commit-ambiguous statuses from the retry surface entirely.
The missing in-loop recheck is pre-existing on the
IsSessionTimeoutpath since #677/#1550. This PR widens what reaches it; the 400 scoping is the mitigation. Tracked separately as #1774 and deliberately not addressed here.Success still depends on the status code alone, so an unreadable body cannot turn a completed purchase into a reported failure — verified by mutation (hoisting the
readErrcheck above the 2xx short-circuit makes the guard fire).Regression tests, both directions
Driven by a body that yields 12 bytes then fails, modeling a response cut short mid-transfer.
TestDoPurchaseTwoStep_TruncatedSessionTimeoutStillRetriesis the regression test for the retry decision, and it is deliberately an attempt-count assertion rather than a message assertion. It fails against the message-only first revision withexpected: 3, actual: 1.This matters beyond this PR. The first revision's test asserted on the error text and passed while the purchase was still abandoned — the assertion was tautological once the message changed, while being labelled as pinning retryability, a property that did not hold. That is the same defect class as #1595 and #1740: an assertion reading as a guarantee it does not provide, written during the very work that was removing them. Only counting attempts distinguishes a changed message from changed behaviour.
The test carries the readable control (which must reach
purchaseMaxAttempts) so it measures the loop rather than a constant; if the control ever stops retrying, the test stops being evidence.TestDoPurchase_UnreadableBodyDoesNotSilentlyLookPermanentnow assertserrors.Is(err, errRetryabilityUnknown)— the thing the loop actually consults — rather than substrings of the message.Two further tests pin the 400 scoping so it cannot be silently widened again, since nothing else in the code would notice:
TestDoPurchase_TruncatedNon400IsNotRetryable— 409, 500, 502 and 504 with truncated bodies must not carry the sentinel.TestDoPurchaseTwoStep_TruncatedNon400DoesNotRetry— a truncated 500 must be attempted exactly once.Both fail when the condition is widened back to any non-2xx, with the message naming the double-buy risk and #1774.
TestDoPurchase_UnreadableBodyOnSuccessStillSucceeds— guards the opposite direction: a truncated body on a 200 must still succeed.Siblings checked, not assumed
The two other
io.ReadAlldiscards in this file are not affected, verified individually::287doCalculatePrice— truncation failsjson.Unmarshal, and an emptyReservationOrderIDis separately guarded.:390fetchReservationOrdersPage— backs the idempotency-token lookup. Truncation failsjson.Unmarshal, and the callerDoIdempotentPurchaseTwoStepexplicitly refuses to purchase on a lookup error rather than falling through, so there is no double-buy path. That code is well guarded.Gates
providers/azure:gofmtclean,go build ./...,go vet ./...,go test -race -count=1 ./...851 passed / 14 packages. Root module: build, vet,gocyclo -over 10 -ignore "_test\.go" .exit 0 empty,golangci-lintv2.10.1 exit 0 with a genuine0 issues.line (both asserted, since a missing--out-formator a concurrent invocation exits 3 with zero findings and reads like a clean run).Rebased onto
6d5275c85.golangci-lintonproviders/azurecompared as a set diff againstorigin/main, not by totals:Drive-by, disclosed: a global spelling fix in this test file also corrected two pre-existing
behaviourmisspellings thatmainalready carried. Two words in a file this PR already rewrites; flagged rather than reverted, since reverting means deliberately restoring a misspelling.The set-diff method earned itself twice here. An intermediate revision removed one finding and added two
misspellin new comments (net 592→593); a later one added onebehaviourthat exactly offset the removederrcheck, leaving the total at 592 — unchanged, and wrong. Totals alone would have passed both times. I also had to re-derive the baseline twice:git checkout -- <path>restores from the index, i.e. my own commit, notmain. The comparison above usesgit checkout origin/main -- providers/azure/.providers/azurecarries 589 pre-existing findings tracked in LeanerCloud/cloud-commitments-platform#174 and LeanerCloud/cloud-commitments-go#63; this PR neither adds to them nor masks them.Known-red inherited from
main:TestGrantCeiling_ConstraintContainmentininternal/authfails onmaintoday (#1737 and #1758 colliding onexecute:ri-exchange). This branch changes onlyproviders/azure/services/internal/reservations/— confirmed bygit diff origin/main --name-only— so that failure is inherited, not caused, and a separate fix is in flight.Summary by CodeRabbit