Background
PR #1225 made the CSV purchase-path orchestration tests assert observable behavior (the written purchase report) instead of assert.NotPanics. That fixed the silent-fixture bug, but every new test in TestRunToolFromCSV* pins ActualPurchase=false (dry-run). The actual-purchase code path -- processPurchaseLoop invoking the service client's PurchaseReservation, adjustRecsForDuplicates hitting DescribeReservedDBInstances / DescribeReservedCacheNodes / DescribeSavingsPlans, real CommitmentID propagation, account-cache hydration -- is currently unverified by cmd/ tests.
Gaps
-
Real-purchase orchestration (ActualPurchase=true). A regression that quietly bypassed the purchase loop on the real path while still writing a dry-run-shaped report would not be caught. Needs an in-memory provider.ServiceClient stub registered via createServiceClient (currently a switch on service) and assertions on:
CommitmentID is non-empty for Success=true rows
Success=false rows carry the error string back through the report
purchaseCount(stub) == len(report-rows-with-Success=true)
-
CSV-replay idempotency. Feeding the same CSV twice through runToolFromCSV with ActualPurchase=true should either suppress duplicates via adjustRecsForDuplicates (current intent) or surface an error -- never double-purchase. Adjacent memory: feedback_upsert_key_matches_hash.md. Needs:
- First run records purchases on the stub.
- Second run with the same CSV sees the duplicate-detection path fire (stub returns the prior purchases as existing RIs).
- Assert second-run report shows zero new purchases OR carries an explicit "already-covered" marker per recommendation.
-
Per-service stub coverage parity. RDS / ElastiCache / OpenSearch / MemoryDB / Redshift each have a createServiceClient branch. At minimum cover RDS + ElastiCache real-purchase paths; OpenSearch / MemoryDB / Redshift can land as paired tests if a stub harness already exists.
Why now
PR #1225's title is "make CSV purchase-path tests assert real behavior". The dry-run-only scope is defensible (it fixes the cited TEST-02 finding), but the broader claim warrants a follow-up that closes the gap on the actual-purchase path. Without it, the next purchase-path orchestration regression has the same silent-failure risk as the original tautological tests.
Triage
P2 / sev medium / impact internal / effort m -- test-only work, no production code change, but blocks a real category of regressions.
Background
PR #1225 made the CSV purchase-path orchestration tests assert observable behavior (the written purchase report) instead of
assert.NotPanics. That fixed the silent-fixture bug, but every new test inTestRunToolFromCSV*pinsActualPurchase=false(dry-run). The actual-purchase code path --processPurchaseLoopinvoking the service client'sPurchaseReservation,adjustRecsForDuplicateshittingDescribeReservedDBInstances/DescribeReservedCacheNodes/DescribeSavingsPlans, realCommitmentIDpropagation, account-cache hydration -- is currently unverified bycmd/tests.Gaps
Real-purchase orchestration (
ActualPurchase=true). A regression that quietly bypassed the purchase loop on the real path while still writing a dry-run-shaped report would not be caught. Needs an in-memoryprovider.ServiceClientstub registered viacreateServiceClient(currently a switch onservice) and assertions on:CommitmentIDis non-empty forSuccess=truerowsSuccess=falserows carry the error string back through the reportpurchaseCount(stub) == len(report-rows-with-Success=true)CSV-replay idempotency. Feeding the same CSV twice through
runToolFromCSVwithActualPurchase=trueshould either suppress duplicates viaadjustRecsForDuplicates(current intent) or surface an error -- never double-purchase. Adjacent memory:feedback_upsert_key_matches_hash.md. Needs:Per-service stub coverage parity. RDS / ElastiCache / OpenSearch / MemoryDB / Redshift each have a
createServiceClientbranch. At minimum cover RDS + ElastiCache real-purchase paths; OpenSearch / MemoryDB / Redshift can land as paired tests if a stub harness already exists.Why now
PR #1225's title is "make CSV purchase-path tests assert real behavior". The dry-run-only scope is defensible (it fixes the cited TEST-02 finding), but the broader claim warrants a follow-up that closes the gap on the actual-purchase path. Without it, the next purchase-path orchestration regression has the same silent-failure risk as the original tautological tests.
Triage
P2 / sev medium / impact internal / effort m -- test-only work, no production code change, but blocks a real category of regressions.