Skip to content

fix(iac/azure): deploy SA cannot create custom reservation role definition (roleDefinitions/write 403) — blocks all Azure deploys #1092

Description

@cristim

Summary

Every recent Deploy to Azure Container Apps run fails in Terraform Apply
while creating the custom reservation role definition. The read-only Azure
Sanity dry run passes, because it never exercises this write path.

Exact error

Error: unexpected status 403 (403 Forbidden) with error: AuthorizationFailed:
The client '***' with object id 'a3e02cbb-a854-4f8a-83a4-6c3388eaefa6' does not
have authorization to perform action
'Microsoft.Authorization/roleDefinitions/write' over scope
'/subscriptions/***/providers/Microsoft.Authorization/roleDefinitions/<guid>'
or the scope is invalid.

  with module.compute_container_apps[0].module.cudly_reservation_role.azurerm_role_definition.cudly_reservation_purchaser,
  on ../../modules/iam/azure/cudly-reservation-role/main.tf line 24
##[error]Terraform exited with code 1.

Reproduced identically across the last ~8 deploy runs (27103373838,
27103282817, 27103152035, 27103067778, 27102839345, 27102621332, 27102222470,
...). Deterministic; only the random role-definition GUID differs per run.

Root cause

The runtime compute module creates a custom Azure role definition:

  • terraform/modules/compute/azure/container-apps/main.tf (~L267) calls
    module "cudly_reservation_role" at subscription scope.
  • terraform/modules/iam/azure/cudly-reservation-role/main.tf L24 declares
    azurerm_role_definition.cudly_reservation_purchaser.

azurerm_role_definition requires Microsoft.Authorization/roleDefinitions/write.
The CI/CD deploy SP's role
(terraform/environments/azure/ci-cd-permissions/locals_data.tf) grants
roleAssignments/{read,write,delete} and roleDefinitions/read only —
roleDefinitions/write is intentionally absent per the CLAUDE.md
bootstrap-vs-runtime IAM split (role-definition creation is bootstrap-only). So
the 403 is the split working as designed; the bug is that a bootstrap-class
resource leaked into the runtime apply path
.

Introduced by PR #744 (f34b036e2), which mirrored the customer-side
reservation role into the host-side container-apps module but copied the role
definition (bootstrap) rather than just the assignment (runtime).

Secondary latent issue in the same module: assignable_scopes includes
/providers/Microsoft.Capacity (tenant root), which a subscription-scoped
principal generally cannot register. Fix together.

Proposed fix (bootstrap, not runtime)

  1. Move module "cudly_reservation_role" into the bootstrap
    terraform/environments/azure/ci-cd-permissions/ (applied by a privileged
    human; has role-admin rights). Expose the role definition ID as an output.
  2. In terraform/modules/compute/azure/container-apps/main.tf, delete the
    module "cudly_reservation_role" block and keep only
    azurerm_role_assignment.reservations_purchaser, resolving the role
    definition via a data.azurerm_role_definition lookup of the pre-created
    role. The deploy SA has roleAssignments/write, so the assignment succeeds.
  3. Drop/correct the /providers/Microsoft.Capacity entry in assignable_scopes.

Do NOT add roleDefinitions/write to the runtime deploy SA — that violates
the documented split ("the apply SHOULD 403 ... that's the signal to re-run the
bootstrap"). The 403 is the intended signal.

Owner approval required (bootstrap IAM is applied manually, staging-first).

Impact

Blocks 100% of Azure Container Apps deploys. Internal/team-only (no end-user
data path), but fully gates the Azure release pipeline.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingeffort/mDaysimpact/internalTeam-internal onlypr-createdA PR has been opened for this issue (dedup guard for the auto-PR loop)pr-mergedThe PR for this issue has been mergedpriority/p1Next up; this sprintseverity/highSignificant harmtriagedItem has been triagedtype/bugDefecturgency/nowDrop other things

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions