Repository navigation
fix(providers): fail loud when all Azure/GCP rec services fail - #1215
Conversation
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 1 minute Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughAzure and GCP recommendation collection now reports total attempted-service failures, while scheduler tests verify failed accounts are not marked successful. Azure service inclusion tracking and related regression coverage were updated; an Azure Terraform file tail also changed. ChangesRecommendation failure guards
Terraform tail update
Estimated code review effort: 4 (Complex) | ~45 minutes Possibly related issues
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
@coderabbitai review |
✅ Action performedReview finished.
|
c78dd67 to
83d25a7
Compare
|
@coderabbitai review |
✅ Action performedReview finished.
|
…tract TestScheduler_CollectAzureRecommendations_Success was asserting require.NoError on a managed_identity account whose credential resolution fails in CI -- the pre-fix shape where every per-service call inside mergeServiceResults silently returned (empty, nil). The COR-03 fix in this PR makes that case fail loud, so the test now fails: "Azure: all 1 accounts failed; last error: ... all 4 Azure recommendation services failed: ...". Rename to _AllAccountsFailLoud and assert the expected error shape end-to-end (per-service all-failed wrap, per-account all-accounts-failed wrap, empty SucceededAccountIDs so the scheduler keeps stale rows). This is exactly the contract the rest of the PR enforces; keeping require.NoError would have regressed the property the fix is trying to add. Adversarial review of #1215.
Adversarial reviewGoal of this PR: when every attempted Azure / GCP recommendation service call errors, surface that as an error to the caller instead of FindingsP0 -- CI was red on a regression introduced by this PR (fixed in 5f3e6a8).
Fix in this push: rename to The remaining P3 / nice-to-have (filed as LeanerCloud/cloud-commitments-go#21).
Things that were right
Minor (not blocking)
Tests run locally on the fix
@coderabbitai full review |
|
✏️ Learnings added
✅ Action performedFull review finished. |
|
@coderabbitai review |
✅ Action performedReview finished.
|
…tract TestScheduler_CollectAzureRecommendations_Success was asserting require.NoError on a managed_identity account whose credential resolution fails in CI -- the pre-fix shape where every per-service call inside mergeServiceResults silently returned (empty, nil). The COR-03 fix in this PR makes that case fail loud, so the test now fails: "Azure: all 1 accounts failed; last error: ... all 4 Azure recommendation services failed: ...". Rename to _AllAccountsFailLoud and assert the expected error shape end-to-end (per-service all-failed wrap, per-account all-accounts-failed wrap, empty SucceededAccountIDs so the scheduler keeps stale rows). This is exactly the contract the rest of the PR enforces; keeping require.NoError would have regressed the property the fix is trying to add. Adversarial review of #1215.
5f3e6a8 to
3709c00
Compare
|
@coderabbitai review |
✅ Action performedReview finished.
|
…tract TestScheduler_CollectAzureRecommendations_Success was asserting require.NoError on a managed_identity account whose credential resolution fails in CI -- the pre-fix shape where every per-service call inside mergeServiceResults silently returned (empty, nil). The COR-03 fix in this PR makes that case fail loud, so the test now fails: "Azure: all 1 accounts failed; last error: ... all 4 Azure recommendation services failed: ...". Rename to _AllAccountsFailLoud and assert the expected error shape end-to-end (per-service all-failed wrap, per-account all-accounts-failed wrap, empty SucceededAccountIDs so the scheduler keeps stale rows). This is exactly the contract the rest of the PR enforces; keeping require.NoError would have regressed the property the fix is trying to add. Adversarial review of #1215.
3709c00 to
e43dc8a
Compare
|
@coderabbitai review |
✅ Action performedReview finished.
|
…tract TestScheduler_CollectAzureRecommendations_Success was asserting require.NoError on a managed_identity account whose credential resolution fails in CI -- the pre-fix shape where every per-service call inside mergeServiceResults silently returned (empty, nil). The COR-03 fix in this PR makes that case fail loud, so the test now fails: "Azure: all 1 accounts failed; last error: ... all 4 Azure recommendation services failed: ...". Rename to _AllAccountsFailLoud and assert the expected error shape end-to-end (per-service all-failed wrap, per-account all-accounts-failed wrap, empty SucceededAccountIDs so the scheduler keeps stale rows). This is exactly the contract the rest of the PR enforces; keeping require.NoError would have regressed the property the fix is trying to add. Adversarial review of #1215.
e43dc8a to
abc9f60
Compare
…tract TestScheduler_CollectAzureRecommendations_Success was asserting require.NoError on a managed_identity account whose credential resolution fails in CI -- the pre-fix shape where every per-service call inside mergeServiceResults silently returned (empty, nil). The COR-03 fix in this PR makes that case fail loud, so the test now fails: "Azure: all 1 accounts failed; last error: ... all 4 Azure recommendation services failed: ...". Rename to _AllAccountsFailLoud and assert the expected error shape end-to-end (per-service all-failed wrap, per-account all-accounts-failed wrap, empty SucceededAccountIDs so the scheduler keeps stale rows). This is exactly the contract the rest of the PR enforces; keeping require.NoError would have regressed the property the fix is trying to add. Adversarial review of #1215.
abc9f60 to
2698cf8
Compare
|
@coderabbitai review |
|
Merge-prep bot: rebased onto main (73166c2). Added fixup commit to correct British English spellings (stabilises->stabilizes, behaviour->behavior) flagged by misspell linter on new lines. Gates: build_azure=0 build_gcp=0 build_root=0; vet_azure=0 vet_gcp=0 vet_scheduler=0; gocyclo=0 (all touched files); lintnew_azure=0 lintnew_gcp=0; test_azure=700 test_gcp=262 test_scheduler=102 (all passed). |
✅ Action performedReview finished.
|
Azure mergeServiceResults and the GCP per-region collection logged per-service errors and dropped them, so a collection where every service or region call failed (expired federated credential, tenant-wide throttle, RBAC gap) returned (recs, nil). The scheduler then counted the account as succeeded, UpsertRecommendations evicted all of its previously collected rows while inserting zero new ones, and last_collection_error was cleared, leaving no signal on the dashboard. Port the AWS 08-H4 all-failed guard to both providers: - Azure: mergeServiceResults now returns an error when every attempted service errored. Services skipped by the params filter carry an attempted flag so a filtered run cannot mask a total failure, and the savingsplans stub (which succeeds unconditionally without any API call) is excluded from the guard so it cannot mask one either. - GCP: collectRegion reports attempted/failed counts plus a representative error, and the new mergeRegionResults fails loud when every attempted (region, service) call errored across all regions. Partial failure keeps the previous tolerated behaviour: any single successful call still returns its recommendations with a nil error. Regression tests cover the all-attempted-failed, partial-failure, filter-skip, savingsplans-stub, and zero-attempt shapes, plus a scheduler-side test asserting a failed collection never lands in SucceededAccountIDs (the eviction eligibility list). Closes #1152
getAdvisorRecommendations swallows pagination errors: when the Azure credential is expired, the auth failure surfaces during pager.NextPage rather than during client construction, so the function always returns (recs, nil) even on a hard auth failure. With advisor carried as attempted=true, the all-attempted-failed guard introduced for COR-03 could never fire on a real total credential failure -- the four real services would each fail (failures=4, attempted=5) and the guard condition failures==attempted would remain false. Fix: set advisor attempted=false, matching the savingsplans exclusion and its rationale. The guard now fires when the four real reservation services (compute, database, cache, cosmosdb) all fail. Update the four guard-behaviour tests to use the production shape (SP=false, advisor=false) and adjust the expected "all N services failed" counts from 5-6 down to 4. Also apply the end-of-file trailing-newline fix pre-commit auto-staged for sp.tf.
…tract TestScheduler_CollectAzureRecommendations_Success was asserting require.NoError on a managed_identity account whose credential resolution fails in CI -- the pre-fix shape where every per-service call inside mergeServiceResults silently returned (empty, nil). The COR-03 fix in this PR makes that case fail loud, so the test now fails: "Azure: all 1 accounts failed; last error: ... all 4 Azure recommendation services failed: ...". Rename to _AllAccountsFailLoud and assert the expected error shape end-to-end (per-service all-failed wrap, per-account all-accounts-failed wrap, empty SucceededAccountIDs so the scheduler keeps stale rows). This is exactly the contract the rest of the PR enforces; keeping require.NoError would have regressed the property the fix is trying to add. Adversarial review of #1215.
Misspell linter (--new-from-rev=origin/main) flagged three British
spellings introduced by the COR-03 commit: "stabilises" in
providers/azure/recommendations.go and "behaviour" in both
providers/azure/recommendations_test.go and
providers/gcp/recommendations_test.go. Changed to American English
("stabilizes", "behavior") to keep the misspell gate clean.
The standalone "Lint Code" CI job runs full golangci-lint on the root module, which caught a British "behaviour" in a comment this PR added at internal/scheduler/scheduler_test.go:1486 (the --new-from-rev provider sweep did not cover the root module). Changed to "behavior" so the root golangci misspell gate is clean.
2698cf8 to
a32b8cd
Compare
|
Merge-prep bot: fixed the Lint Code CI failure. The standalone "Lint Code" job runs full golangci-lint on the root module (not incremental), which caught a British "behaviour" in a comment this PR added at internal/scheduler/scheduler_test.go:1486; the earlier --new-from-rev provider sweep did not cover the root module. Changed to "behavior". Grepped the entire PR diff for other British spellings the misspell linter catches (behaviour/colour/optimise/stabilise/cancelled/catalogue/honour/etc.): none remain in added lines. Full-suite verification matching the CI Lint Code job (true exit codes, no pipe masking):
Rebased onto latest main (d644f11). Diff intact: 6 files, +358/-40. |
|
Merged to main (rebased onto current main, CLEAN + all CI green; closes #1152). fail loud when ALL Azure/GCP recommendation services fail instead of returning an empty success (no-silent-fallback on the money-adjacent recs path). |
Problem
Closes #1152 (review finding COR-03).
Azure's
mergeServiceResultsand GCP's per-region collection logged per-service errors at WARN and dropped them, so a collection where every service or region call failed (expired federated credential, tenant-wide throttle, RBAC gap) returned(recs, nil). Downstream,fanOutPerAccountcounted the account as succeeded,UpsertRecommendationsevicted all previously collected rows for it while inserting zero new ones, andlast_collection_errorwas cleared, so the dashboard silently lost the account's recommendations with no error banner. AWS already had a guard for exactly this hazard (the 08-H4 all-failed guard inproviders/aws/recommendations/client.go).Fix
Port the AWS 08-H4 guard to both providers:
providers/azure/recommendations.go):mergeServiceResultsnow returns([]common.Recommendation, error)and fails loud when every attempted service errored.serviceResultgains anattemptedflag so services skipped by the params filter are not counted as successes. The savings plans client is a stub that succeeds unconditionally without making any API call, so it is excluded from the guard; otherwise the guard could never fire on a total provider failure. A note on the stub documents flipping the flag back when the Benefits Recommendations API stabilises.providers/gcp/recommendations.go):collectRegionnow reports attempted/failed call counts plus a representative error inregionResult, and the newmergeRegionResultsfails loud when every attempted (region, service) call errored across all regions.Partial failure keeps the previous tolerated behaviour: one successful call is enough to return the successful results with a nil error, with failures logged at WARN as before. With the error propagating, the scheduler counts the account as failed, keeps its stale rows, records
last_collection_error, and the freshness banner surfaces.Tests
providers/azure:TestMergeServiceResults_AllAttemptedFailed,_SkippedServicesDoNotMaskTotalFailure,_SavingsPlansStubDoesNotMaskTotalFailure,_PartialFailureStillSucceeds, plus the updated_OrderIsStable.providers/gcp:TestMergeRegionResults_AllAttemptedFailed,_PartialFailureStillSucceeds,_NoAttemptsIsNotAFailure.internal/scheduler:TestFanOutPerAccount_FailedCollectionNotInSucceededAccountIDspins that a failed collection is never eligible for stale-row eviction.Ran
go build ./...plusgo vet/go test ./...in the root module'sinternal/schedulerpackage (102 passed) and in theproviders/azure(700 passed) andproviders/gcp(262 passed) modules. The new merge-level regression tests cannot compile against the pre-fix code (the pre-fixmergeServiceResultsreturned only a slice andmergeRegionResultsdid not exist), which is the structural proof they exercise the new fail-loud contract.Summary by CodeRabbit