Summary
Every recent Deploy to Azure Container Apps run fails in Terraform Apply
while creating the custom reservation role definition. The read-only Azure
Sanity dry run passes, because it never exercises this write path.
Exact error
Error: unexpected status 403 (403 Forbidden) with error: AuthorizationFailed:
The client '***' with object id 'a3e02cbb-a854-4f8a-83a4-6c3388eaefa6' does not
have authorization to perform action
'Microsoft.Authorization/roleDefinitions/write' over scope
'/subscriptions/***/providers/Microsoft.Authorization/roleDefinitions/<guid>'
or the scope is invalid.
with module.compute_container_apps[0].module.cudly_reservation_role.azurerm_role_definition.cudly_reservation_purchaser,
on ../../modules/iam/azure/cudly-reservation-role/main.tf line 24
##[error]Terraform exited with code 1.
Reproduced identically across the last ~8 deploy runs (27103373838,
27103282817, 27103152035, 27103067778, 27102839345, 27102621332, 27102222470,
...). Deterministic; only the random role-definition GUID differs per run.
Root cause
The runtime compute module creates a custom Azure role definition:
terraform/modules/compute/azure/container-apps/main.tf (~L267) calls
module "cudly_reservation_role" at subscription scope.
terraform/modules/iam/azure/cudly-reservation-role/main.tf L24 declares
azurerm_role_definition.cudly_reservation_purchaser.
azurerm_role_definition requires Microsoft.Authorization/roleDefinitions/write.
The CI/CD deploy SP's role
(terraform/environments/azure/ci-cd-permissions/locals_data.tf) grants
roleAssignments/{read,write,delete} and roleDefinitions/read only —
roleDefinitions/write is intentionally absent per the CLAUDE.md
bootstrap-vs-runtime IAM split (role-definition creation is bootstrap-only). So
the 403 is the split working as designed; the bug is that a bootstrap-class
resource leaked into the runtime apply path.
Introduced by PR #744 (f34b036e2), which mirrored the customer-side
reservation role into the host-side container-apps module but copied the role
definition (bootstrap) rather than just the assignment (runtime).
Secondary latent issue in the same module: assignable_scopes includes
/providers/Microsoft.Capacity (tenant root), which a subscription-scoped
principal generally cannot register. Fix together.
Proposed fix (bootstrap, not runtime)
- Move
module "cudly_reservation_role" into the bootstrap
terraform/environments/azure/ci-cd-permissions/ (applied by a privileged
human; has role-admin rights). Expose the role definition ID as an output.
- In
terraform/modules/compute/azure/container-apps/main.tf, delete the
module "cudly_reservation_role" block and keep only
azurerm_role_assignment.reservations_purchaser, resolving the role
definition via a data.azurerm_role_definition lookup of the pre-created
role. The deploy SA has roleAssignments/write, so the assignment succeeds.
- Drop/correct the
/providers/Microsoft.Capacity entry in assignable_scopes.
Do NOT add roleDefinitions/write to the runtime deploy SA — that violates
the documented split ("the apply SHOULD 403 ... that's the signal to re-run the
bootstrap"). The 403 is the intended signal.
Owner approval required (bootstrap IAM is applied manually, staging-first).
Impact
Blocks 100% of Azure Container Apps deploys. Internal/team-only (no end-user
data path), but fully gates the Azure release pipeline.
Summary
Every recent
Deploy to Azure Container Appsrun fails inTerraform Applywhile creating the custom reservation role definition. The read-only Azure
Sanity dry run passes, because it never exercises this write path.
Exact error
Reproduced identically across the last ~8 deploy runs (27103373838,
27103282817, 27103152035, 27103067778, 27102839345, 27102621332, 27102222470,
...). Deterministic; only the random role-definition GUID differs per run.
Root cause
The runtime compute module creates a custom Azure role definition:
terraform/modules/compute/azure/container-apps/main.tf(~L267) callsmodule "cudly_reservation_role"at subscription scope.terraform/modules/iam/azure/cudly-reservation-role/main.tfL24 declaresazurerm_role_definition.cudly_reservation_purchaser.azurerm_role_definitionrequiresMicrosoft.Authorization/roleDefinitions/write.The CI/CD deploy SP's role
(
terraform/environments/azure/ci-cd-permissions/locals_data.tf) grantsroleAssignments/{read,write,delete}androleDefinitions/readonly —roleDefinitions/writeis intentionally absent per the CLAUDE.mdbootstrap-vs-runtime IAM split (role-definition creation is bootstrap-only). So
the 403 is the split working as designed; the bug is that a bootstrap-class
resource leaked into the runtime apply path.
Introduced by PR #744 (
f34b036e2), which mirrored the customer-sidereservation role into the host-side container-apps module but copied the role
definition (bootstrap) rather than just the assignment (runtime).
Secondary latent issue in the same module:
assignable_scopesincludes/providers/Microsoft.Capacity(tenant root), which a subscription-scopedprincipal generally cannot register. Fix together.
Proposed fix (bootstrap, not runtime)
module "cudly_reservation_role"into the bootstrapterraform/environments/azure/ci-cd-permissions/(applied by a privilegedhuman; has role-admin rights). Expose the role definition ID as an output.
terraform/modules/compute/azure/container-apps/main.tf, delete themodule "cudly_reservation_role"block and keep onlyazurerm_role_assignment.reservations_purchaser, resolving the roledefinition via a
data.azurerm_role_definitionlookup of the pre-createdrole. The deploy SA has
roleAssignments/write, so the assignment succeeds./providers/Microsoft.Capacityentry inassignable_scopes.Do NOT add
roleDefinitions/writeto the runtime deploy SA — that violatesthe documented split ("the apply SHOULD 403 ... that's the signal to re-run the
bootstrap"). The 403 is the intended signal.
Owner approval required (bootstrap IAM is applied manually, staging-first).
Impact
Blocks 100% of Azure Container Apps deploys. Internal/team-only (no end-user
data path), but fully gates the Azure release pipeline.