ci(azure): Deploy to Azure Container Apps has failed on every main run for 3 weeks — no Azure change since 2026-07-19 is deployed

[cristim]

The Azure deploy has not succeeded on main in the last 100 workflow runs, going back to at least 2026-07-19. Found while checking whether #1223 was stale.

Deploy to Azure Container Apps, branch=main, last 100 runs:
  failure:   68
  cancelled: 32
  success:    0
  oldest run in window: 2026-07-19

It fails at Terraform Apply, not at build or test:

Error: authorization.RoleAssignmentsClient#Create: Failure responding ...
Error: loading Role Definition List: could not find role 'CUDly Reservation Purchaser (custom) - ***'

Blocked resources: azurerm_role_assignment.cost_management_reader and azurerm_role_assignment.subscription_reader.

Why this went unnoticed for three weeks

CI - Build & Test is green on main and has been across every recent merge (a37e14790, d6e60f637, 726389b48). Every PR gate we look at is green. The deploy workflow is a separate run that nothing surfaces, so "CI is green" has been true and irrelevant at the same time.

This is the same shape as the multi-module gap in #1751: a check that looked comprehensive was scoped to something narrower than anyone assumed. There, ./... covered one module of six. Here, "CI green" covers build and test but not deploy.

What it means for what we have been merging

If this workflow is what ships the running Azure service, then no Azure change merged since 2026-07-19 is live. That window includes several providers/azure money-path fixes merged in the last day:

• fix(azure): surface the read error that decides purchase retry-vs-abort #1767 — purchase retry no longer abandons on a truncated "Session timed out" body
• fix(azure): pair the recommended SKU with the count in its own units #1771 — Legacy/EA recommendations no longer pair the normalized SKU with the un-normalized quantity
• sec(azure): re-check idempotency between in-process purchase retries #1793 — idempotency re-checked between in-process retries, preventing a double purchase
• sec(auth): carve execute:ri-exchange out of admin:* and seed the granting group #1758 — execute:ri-exchange carved out of admin:*

Verify that inference before acting on it. I have established the workflow fails; I have not established what it deploys, whether another path ships the Azure service, or whether the failing role assignments are the last step or an early one that aborts before anything useful happens. A partial apply that fails at the end has different consequences from one that fails before deploying the app image.

The immediate cause

A custom role definition named CUDly Reservation Purchaser (custom) - <subscription> does not exist in the target subscription, so Terraform cannot resolve it to assign it.

Establish which of these it is before fixing:

1. The role was never created in this subscription (bootstrap step missed or a new subscription).
2. The role was created and has since been deleted or renamed.
3. The Terraform name template drifted from what the bootstrap creates (a suffix, a casing change, an interpolation that renders differently).
4. The deploy identity lacks permission to read role definitions, so the role exists but the lookup returns nothing. This one looks like a missing role but is a permissions failure, and the error text does not distinguish it.

Option 4 is worth ruling out explicitly: Microsoft.Authorization/roleDefinitions/read is a separate grant from the ability to create assignments, and a caller without it sees an empty list rather than a permission error.

Verification for whatever lands

• A deploy run on main reaching success, not merely getting past this error to a different one.
• The two blocked role assignments present in the subscription afterwards.
• Confirm the deployed image actually corresponds to current main, since three weeks of failed applies may have left state that a single success does not fully reconcile.

Related

• Main CI red post-#1220: golangci-lint flood, npm audit, internal/config integration failures #1223 — separately confirmed stale and being closed; CI - Build & Test is green on main.
• ci: unit and integration test jobs only run the root module — providers/*, pkg and tests/e2e are never tested #1751 — same class of blind spot: a green check whose scope was narrower than assumed.

[cristim]

Corrected root cause: the bootstrap role is unappliable, not unapplied

This issue currently says the custom role was never created because the bootstrap step was missed. That is wrong, and the real cause has a wider blast radius. Established by execution against the live subscription, not by reading.

The actual cause

Microsoft.Capacity/reservationOrders/purchase/action does not exist in Azure's operation catalog.

Applying terraform/environments/azure/ci-cd-permissions fails with:

Error: unexpected status 400 (400 Bad Request) with error: InvalidActionOrNotAction:
'Microsoft.Capacity/reservationOrders/purchase/action' does not match any of the
actions supported by the providers.

  with module.cudly_reservation_role.azurerm_role_definition.cudly_reservation_purchaser,
  on ../../../modules/iam/azure/cudly-reservation-role/main.tf line 35

Azure rejects the whole role definition over that one entry, so the role is never created, so the deploy stack's name-based data.azurerm_role_definition lookup finds nothing. The bootstrap stack was not skipped: it cannot succeed as written, and re-running it would never have helped.

I validated all 12 actions in the module against the live catalog (az provider operation show, covering both namespace-level operations[] and resourceTypes[].operations[]). Exactly one is invalid; the other 11 are fine.

Hypothesis tested and ruled out

Microsoft.Capacity and Microsoft.BillingBenefits were both NotRegistered in subscription 24d185cc. That looked like the answer, since Azure validates role actions against registered providers. I registered both and re-applied: identical error. Registration was not the cause.

The two registrations are additive, reversible, and arguably correct to keep, but they were an incidental change rather than the fix, and I am flagging them rather than leaving them undisclosed.

The module's stated premise is also wrong

The module justifies its own existence with:

The built-in Reservation Purchaser role lacks reservationOrders/purchase/action, which causes 403 on production reservation purchases.

The built-in role's actual actions do not include Microsoft.Capacity/reservationOrders/write, Microsoft.Capacity/calculatePrice/action, or any Microsoft.BillingBenefits action. So the custom role's purpose is legitimate, but the action added to serve that purpose is fictional. Purchasing a reservation is PUT /providers/Microsoft.Capacity/reservationOrders/{id}, which is reservationOrders/write and was already in the list.

Blast radius is wider than this issue

arm/CUDly-CrossSubscription/template.json:51 carries the same invalid action. That template is customer-facing: customers deploy it to grant CUDly cross-subscription access. It should fail identically for every customer who has run it. That deserves its own triage.

iac/federation/azure-target/terraform is the other customer-side consumer.

Why no test caught this

scripts/testdata/role-parity/matching-arm.json and matching-tf.tf.fixture both contain the invalid action. The ARM-vs-TF parity checker therefore passed while both sides were invalid. The fixtures prove the two representations agree; nothing proved either is a thing Azure accepts. Parity is not validity.

Verification

Removing the single invalid action and applying against the live subscription created the role:

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
reservation_role_definition_id = "/subscriptions/24d185cc-.../roleDefinitions/ab9419be-..."

Confirmed in both directions: fails with the action present, succeeds with it removed.

Answering the open question about deployed state

The container app is running cudly:ddeeb98-20260811004918, and ddeeb98 is the current main HEAD. The deploy job builds and ships the image, then fails later during terraform apply when the role lookup is resolved (plan defers it: the log shows data.azurerm_role_definition.cudly_reservation_purchaser will be read during apply).

So application code from the merged Azure money-path work is live. What has not been applying is the Terraform-managed infrastructure ordered after the failure point. That is a meaningfully better position than the worst case, but the two should not be conflated.

A rerun of the failed deploy is in flight now that the role exists; I will post the result.

Note on the CI diagnostic from #1795

The Explain a missing bootstrap role step added in #1795 tells operators the bootstrap stack was never applied. Given the above, that message points at the wrong cause and is being corrected in the fix PR.