Reservation-purchaser role assignment is destroyed and recreated on every apply, briefly revoking a money-path grant

[cristim]

module.compute_container_apps[0].azurerm_role_assignment.reservations_purchaser is destroyed and recreated on every single apply, so the host identity's reservation-purchase grant is briefly revoked on every deploy.

Found while verifying the #1794 fix in deploy run 31545175691, where it appears as one of exactly three destroys.

Why

Both role_definition_id and scope derive from a data source that is deferred to apply time (data.azurerm_role_definition.cudly_reservation_purchaser, which Terraform reports as will be read during apply). Both attributes are therefore always (known after apply), and both force replacement on azurerm_role_assignment. So the resource can never be seen as unchanged, regardless of whether anything actually changed.

Why it matters

This is a money-path grant. It is what lets the host identity call calculatePrice and reservationOrders/write. There is a window on every deploy where the assignment does not exist, and a purchase attempting to run in that window would 403.

The window is short, but the deploy is not coordinated with purchase activity in any way, so nothing rules the overlap out. It also produces misleading churn in every plan, which trains reviewers to skim past a destroy line on a permissions resource, and that is its own hazard.

Fix direction

Stop making the assignment depend on apply-time-unknown values. Options worth weighing:

• Resolve the role definition by a stable, statically-known ID rather than a name lookup, so role_definition_id is known at plan time. The bootstrap stack already outputs role_definition_resource_id; consuming that through a documented channel (remote state or an explicit variable) removes the data source entirely.
• Failing that, pin scope to a statically-derivable subscription ID rather than routing it through the deferred data source.

Note the deliberate bootstrap-vs-runtime split documented in terraform/modules/compute/azure/container-apps/main.tf: the role definition is created by the human-applied bootstrap stack, and the runtime stack only looks it up. Any fix has to respect that split rather than move definition ownership into the runtime stack.

Verification

Apply twice with no intervening changes and confirm the second plan reports no changes for this resource. That is the whole test, and today it fails.

Then confirm the grant is continuously present across a deploy rather than merely present afterwards, since an assignment that is destroyed and recreated still ends in the correct final state while having been absent in between.

Related

• ci(azure): Deploy to Azure Container Apps has failed on every main run for 3 weeks — no Azure change since 2026-07-19 is deployed #1794 / PR fix(iac/azure): remove nonexistent Microsoft.Capacity action from reservation-purchaser role #1800 - the role-definition fix that made this visible, by letting apply reach this resource at all.
• Azure deploy silently loses Terraform state: branch-keyed concurrency plus an unconditional lease break #1801 - separate state-integrity defect in the same pipeline.

[cristim]

Correction: this affects all three subscription-scoped role assignments, not just the purchaser

I filed this naming only reservations_purchaser. That was too narrow, and the cause I gave was a symptom rather than the root.

Verified against the apply log of run 31545175691 (the first green Azure deploy since 2026-07-19):

azurerm_role_assignment.cost_management_reader: Destroying / Creating
azurerm_role_assignment.subscription_reader:    Destroying / Creating
azurerm_role_assignment.reservations_purchaser: Destroying / Creating

All three are destroyed and recreated on every apply.

Actual root cause

module "compute_container_apps" carries a module-level depends_on (terraform/environments/azure/compute.tf:75) that targets module.build among others, and module.build changes on every deploy. Terraform propagates a module-level depends_on to every data source in the module, so both module data sources are deferred to apply time. That makes scope (from data.azurerm_subscription.current) (known after apply) for all three assignments, and role_definition_id additionally unknown for the purchaser.

So the purchaser is replaced on two unknown attributes and the other two on one. My original diagnosis identified only the second of those, which is why I named only one resource.

Why the wider scope matters

The revocation window on each deploy covers:

• Cost Management Reader - the Consumption API access behind reservation recommendations and reservation detail reads
• Reader - host-subscription enumeration for VMs, Redis, Cosmos, Search, SQL and compute SKUs when the host account is ingested as "Self"
• the custom reservation-purchaser role - calculatePrice and reservationOrders/write, the money path

That is not one grant briefly flickering, it is the app's entire host-subscription read and purchase capability. Anything running during a deploy can 403 on any of the three.

I am leaving the priority at p2 rather than raising it unilaterally, because the window is short and no incident is attributed to it. Anyone who disagrees given the wider blast radius should re-triage it explicitly rather than quietly working around the label.

Note on the fix

PR #1804 resolves both inputs in the caller, so it covers all three assignments rather than only the purchaser. It deliberately does not delete the module depends_on, which is the true root cause and a one-line change, because that depends_on also orders the container app after resources no referenced output covers (NSG associations, jwt/session secrets, DB firewall rules). Removing it changes greenfield-apply ordering, which is a larger change than this issue. That tradeoff is recorded in the PR.

Still unverified

The real test is two consecutive live applies with no intervening change, with the second plan reporting no changes for these resources. That has not run. terraform validate passing is not evidence for this.

[cristim]

Verified in production: the churn has stopped

PR #1804 shipped with this explicitly unverified, because the real test needs two consecutive live applies and could not be run from a developer machine. Those applies have now happened.

Before the fix - run 31545175691 (head 51e89a6, pre-#1804):

azurerm_role_assignment.cost_management_reader: Destroying / Creating
azurerm_role_assignment.subscription_reader:    Destroying / Creating
azurerm_role_assignment.reservations_purchaser: Destroying / Creating

After the fix - two consecutive successful deploys, both with module.compute_container_apps in scope:

run	head	role assignments refreshed	destroyed / created / modified
31574048862	2c87db0	10	0
31576483437	8b16ed4	10	0

Every one of the ten role assignments appears only as Refreshing state.... None is destroyed, created or modified in either run. The money-path grant (calculatePrice + reservationOrders/write) is no longer revoked on each deploy, and neither are Cost Management Reader or Reader.

Limits of this evidence, stated plainly

gh run view --log appears to truncate: neither captured log contains an Apply complete! line or a Plan: summary, so this is not a plan-summary-level proof of "no changes". What it is: two post-fix applies in which none of the ten assignments shows a mutation verb, against a pre-fix apply in which three of them show Destroying and Creating explicitly.

That is a real before/after signal on the same log source and the same extraction, and both runs concluded success across all four jobs. It is strong indicative evidence rather than the definitive plan diff. Anyone wanting the stricter proof should read the Terraform Plan step output of a future run directly and confirm it reports no changes for module.compute_container_apps[0].azurerm_role_assignment.reservations_purchaser.

One prediction from the PR that held

PR #1804 noted that if the AZURE_SUBSCRIPTION_ID secret's casing differed from Azure's canonical lowercase, the first plan after the change would still show one final replacement before converging, and that lower() is what makes it converge rather than churn forever. Two consecutive clean runs are consistent with convergence having already happened.