fix(ci/azure): serialize Terraform state writers on environment, not branch

[cristim]

Closes #1801

What was broken

The Azure Terraform state blob is keyed on environment (github-<env>.terraform.tfstate), but deploy-azure.yml serialized on branch (concurrency.group: deploy-azure-${{ github.ref }}). prepare maps every non-dispatch event to dev, so two runs on different refs landed in different concurrency groups, ran simultaneously, and applied against one state file.

On its own that collision would have been loud: the Azure blob lease makes the second writer fail with Error acquiring the state lock. Three az storage blob lease break invocations removed that safety net. None of them checked that a lease was held, none checked its age, and none passed --lease-break-period, so the default broke an active lease immediately. The step named "Break stale state lock" never established staleness. That is what turned a noisy collision into a silent one: the second run broke the first's live lease, both wrote, and the later writer's stale read dropped the earlier one's entries.

The fix

1. Serialize on the value that identifies the state file.

The trap here is that workflow-level concurrency cannot see needs; it is limited to github, inputs and vars. Rather than re-deriving the environment at that level (which would duplicate prepare's mapping and, on the workflow_call path from deploy-all.yml, risk not applying at all), the group moves to the job level, where needs is in scope:

# deploy-azure.yml, job build-and-deploy
concurrency:
  group: azure-tfstate-${{ needs.prepare.outputs.environment }}
  cancel-in-progress: false

This is the exact value the next step interpolates into the state key, so the group and the state file cannot drift apart. Workflow-level concurrency is removed; it was fully subsumed (same ref implies same environment implies same group) and keeping it would have implied a state guard it never provided.

The same group is applied to the other two writers of that blob, so serialization holds across workflows, not just within one:

file	job	state key	group
deploy-azure.yml	build-and-deploy	github-<env>.terraform.tfstate	azure-tfstate-<env>
cleanup-staging.yml	destroy-azure	github-staging.terraform.tfstate	azure-tfstate-staging
rollback.yml	rollback-azure	github-<env>.terraform.tfstate	azure-tfstate-<env>

deploy-all.yml is the fourth workflow named in the issue and gets no group, only a comment saying so. It reaches the state exclusively through uses: ./.github/workflows/deploy-azure.yml, whose build-and-deploy job runs as a real job of that run and is serialized there. Putting the same group on the caller job would deadlock: the caller would hold the group while waiting on the inner job queued behind it.

cancel-in-progress: false is stated explicitly on all three rather than left to the default. Cancelling mid-terraform apply is how you get a half-applied stack and a lease nobody releases.

2. Delete the lease breaks, all three.

The two pre-emptive ones (deploy-azure.yml, cleanup-staging.yml) are the defect itself. Each step also generated /tmp/backend.tfbackend, which terraform init consumes, so the step survives as a renamed backend-config writer with the lease break removed.

The third, Release state lock on failure (if: failure() || cancelled()), is deleted rather than made conditional. The decisive argument is the cancelled() half: GitHub SIGINTs the step and then runs the if: cancelled() steps while terraform apply is still gracefully shutting down and may still be writing state, so the step broke a live lease out from under an active writer. That is a corruption vector, not a cleanup.

An earlier draft of this section claimed the failure() half was simply a no-op because Terraform always unlocks on its way out. Adversarial review showed that overstates it, and the code comment has been corrected. A lease is genuinely stranded when:

• the unlock call itself fails (Error releasing the state lock, e.g. OIDC token expiry mid-apply, a 403 on the storage account), and those conditions correlate with whatever failed the apply, so this is not exotic;
• Terraform crashes;
• Actions SIGKILLs after the cancellation grace period rather than letting the graceful shutdown finish.

The azurerm backend takes an infinite blob lease, so none of these self-expire. That residue is accepted rather than denied: recovery is terraform force-unlock <ID> with the ID Terraform prints. The reason to delete anyway is that a step which clears those cases can only do so by also breaking live leases, which is the defect this issue is about. A loud Error acquiring the state lock is the correct outcome of a real collision and is now allowed to happen.

How this would have prevented the June 9 incident

Per #1801, at the creation instant of cost_management_reader (2026-06-09T15:42:51Z):

run	branch	window	old group	new group
27217251153	feat/multicloud-web-frontend	15:32:24 -> 15:58:15	deploy-azure-refs/heads/feat/multicloud-web-frontend	azure-tfstate-dev
27217600951	main	15:38:01 -> 16:03:01	deploy-azure-refs/heads/main	azure-tfstate-dev

Both runs are non-dispatch events, so prepare returned dev for both and both wrote github-dev.terraform.tfstate. Under the old group they were in different groups and both ran. Under the new group they are in the same group: run 27217600951's build-and-deploy would have sat pending at 15:38:01 until 27217251153 finished at 15:58:15, and would then have terraform init'd against the state that run had just written. There is no window in which both hold the blob, so no read-modify-write of a stale state, and no dropped role assignments.

Both runs were verified as event: push via gh run view, so both took prepare's *) branch. Note the feature-branch run was possible because deploy-azure.yml reached pushes beyond main at the time; today's on.push.branches: [main] narrows that particular path. It does not close the hole: a workflow_dispatch can still be launched from any ref, and deploy-all.yml, rollback.yml and cleanup-staging.yml all reach the same blob independently of the push trigger. The trigger list is not the guard; the concurrency group is.

Belt and braces: even if the groups were somehow bypassed, deleting the lease breaks restores the blob lease as a real barrier, so the second writer fails loudly instead of proceeding.

Verification

command	exit	result
actionlint deploy-azure.yml deploy-all.yml cleanup-staging.yml rollback.yml	1	268 lines of output, byte-identical to the same command on origin/main after normalizing file:line prefixes. Every finding is a pre-existing shellcheck SC2086/SC2129 info/style note on steps this PR does not touch. Zero new findings. actionlint is not wired into CI or pre-commit, so this is not a gate either way.
negative control: needs.… added to a workflow-level concurrency in a scratch copy	1	context "needs" is not allowed here. available contexts are "github", "inputs", "vars"
the same expression at job level (this PR)	n/a	no such error

The negative control matters: it proves actionlint actually enforces context availability for concurrency, so the clean run on the job-level form is evidence the expression resolves, not evidence that nothing was checked. It also confirms the docs' scoping rule first-hand rather than on trust.

prepare's environment derivation was re-executed standalone for every trigger path to confirm the group can never be partially empty:

event=push              input=''        -> key=github-dev.terraform.tfstate      group=azure-tfstate-dev
event=push (other ref)  input=''        -> key=github-dev.terraform.tfstate      group=azure-tfstate-dev
event=workflow_dispatch input='dev'     -> key=github-dev.terraform.tfstate      group=azure-tfstate-dev
event=workflow_dispatch input='staging' -> key=github-staging.terraform.tfstate  group=azure-tfstate-staging
event=workflow_dispatch input='prod'    -> key=github-prod.terraform.tfstate     group=azure-tfstate-prod
event=workflow_call     input='staging' -> key=github-staging.terraform.tfstate  group=azure-tfstate-staging
event=workflow_call     input=''        -> prepare FAILS, build-and-deploy skipped, group never evaluated
event=workflow_call     input='bogus'   -> prepare FAILS, build-and-deploy skipped, group never evaluated

The rows are labelled by the EVENT_NAME value fed to prepare, since that is what its script branches on. Note per #1805 that a real workflow_call never presents workflow_call as github.event_name; that does not change the empty-suffix conclusion, because every arm of the first case statement is still filtered by the allowlist below.

The load-bearing property: can the group ever be azure-tfstate- with an empty suffix?

No, and importantly the *) ENVIRONMENT=dev ;; catch-all is not what guarantees it. That arm only covers non-dispatch events. The workflow_dispatch|workflow_call arm assigns ENVIRONMENT="$INPUT_ENVIRONMENT", which prepare explicitly defaults to the empty string (INPUT_ENVIRONMENT="${INPUT_ENVIRONMENT:-}") and which is typed as a free-form string on the workflow_call path. So an empty value is reachable inside the step.

What makes it safe is the second case statement, the allowlist:

case "$ENVIRONMENT" in
  dev|staging|prod) ;;
  *) echo "::error::Refusing unknown environment: $ENVIRONMENT"; exit 1 ;;
esac

Empty string does not match dev|staging|prod, so it falls to *) and exits 1 under set -euo pipefail. prepare then fails, and build-and-deploy declares needs: prepare with no if: (verified by reading the job's direct children: name, runs-on, needs, concurrency, outputs, env, steps, and no if). A job whose needs failed or was skipped is skipped, so it never dispatches and never occupies a group. Job-level concurrency is evaluated at dispatch, after needs resolves, so there is no ordering in which an empty value claims a group.

Verified empirically, not inferred: the workflow_call input='' and input='bogus' rows in the table above are actual executions of prepare's script, and both exit non-zero.

The other load-bearing property: do the three writers share one vocabulary?

They only serialize against each other if they produce the identical string for the same environment. A development here against a dev there would put a rollback and a deploy in different groups against one blob, which is the original bug on the worst possible path.

writer	source of the value	possible values
deploy-azure.yml build-and-deploy	needs.prepare.outputs.environment	dev, staging, prod only, enforced by prepare's runtime allowlist (it needs one, because workflow_call inputs are free-form strings)
rollback.yml rollback-azure	inputs.environment	dev, staging, prod only, enforced by type: choice, options: [dev, staging, prod] at rollback.yml:34-38. rollback.yml has only a workflow_dispatch trigger (no workflow_call, no release), so there is no free-form path into it
cleanup-staging.yml destroy-azure	literal	staging

Same three lowercase tokens, no divergence reachable. (GitHub also documents concurrency group names as case insensitive, so a future case slip would not split a group either.)

And each group is keyed on the same value its own state key is built from, which is what stops this fix reproducing the original defect somewhere new:

file	group	state key	same value?
deploy-azure.yml	:124 needs.prepare.outputs.environment	:161 ENVIRONMENT: needs.prepare.outputs.environment -> :163 github-%s.terraform.tfstate	yes
rollback.yml	:418 inputs.environment	:473 ENVIRONMENT: inputs.environment -> :477 github-%s.terraform.tfstate	yes
cleanup-staging.yml	:260 literal staging	:303 literal github-staging.terraform.tfstate	yes

All three resolve to the same blob name shape github-<env>.terraform.tfstate. (rollback.yml:234 and :322 use github-%s/terraform.tfstate with a slash, but those are the AWS jobs on a different backend, correctly outside this group.)

Repo-wide check for other writers: rg across .github/workflows/, scripts/ and Makefile finds no other CI path that runs terraform apply/destroy against terraform/environments/azure. database-migration.yml's migrate-azure job runs only terraform output, which takes no lock, so it needs no group.

What remains unproven

• The concurrency behaviour itself is not empirically demonstrated. Two overlapping live runs were not staged, and the group semantics can only be observed on GitHub's runners. The evidence here is that the expression is valid, in scope, and evaluates to the intended string on every path, plus that the vocabularies align. It is not evidence that GitHub queued anything. This is the single largest gap.
• A queued rollback or destroy can be evicted. GitHub keeps one pending entry per group; cancel-in-progress: false protects the running job, not the pending one. A rollback-azure queued behind an in-flight prod deploy is cancelled if another deploy queues into the same group. A rollback evaporating mid-incident is a real cost of sharing one group across three workflows, and it is inherent to that sharing. It is not silent: rollback.yml's summary job exits 1 on any non-success result, so an evicted rollback reddens the run. cleanup-staging.yml has no such aggregator, so an evicted destroy-azure shows only as a cancelled job. Documented in the job comments.
• The environment: bindings are a hazard, not just an unknown. Beyond the open question of whether an approval-parked job holds its group, if it does, the mitigation with the right shape is the one already used for deploy-all.yml: keep the group off the approval-gated job and put it on an inner job past the gate. Not done here because no environment currently carries protection rules (#1660) and the restructure is larger than this fix warrants.
• The first real cross-workflow collision after this merges is the actual test. The first push to main will exercise the deploy path; nothing will exercise the cleanup-staging.yml / rollback.yml groups until one of those workflows is dispatched.
• Nothing enforces the group prefix staying in sync. The three azure-tfstate- literals are matched by convention. A future writer of that blob added without the group, or a typo in the prefix, would silently not serialize, the same class of silent failure as the original bug. Filed as a follow-up so it does not evaporate.
• Interaction with the environment: bindings is untested. rollback-azure and destroy-azure are bound to deployment environments. If required-reviewer rules are ever configured on those (per sec(ci): deployment environments have no protection rules, so environment-bound credentialed jobs run unapproved cloud-commitments-platform#141 none exist today), it is unclear to me whether a job parked awaiting approval holds its concurrency group. If it does, an unapproved rollback would block deploys to that environment until someone acts on it. I could not settle this from the docs and did not test it. Blocking is still the safer failure than concurrent writers, so it does not change the design, but it is a real unknown.
• The lease-break deletion trades a self-healing path for a loud one. A Terraform crash that strands a lease now wedges the pipeline until someone runs force-unlock. That is the intended trade per the issue, but it is a real operational change, not a free win.

Other providers: BOTH halves of this bug are live on AWS and GCP

An earlier draft of this section claimed the silent-loss half did not transfer to AWS because S3 uses a lock file rather than a blob lease. That was wrong, and adversarial review caught it. Corrected:

Half 1, branch-keyed groups against environment-keyed state:

• deploy-aws-lambda.yml:33-35 -> deploy-lambda-${{ github.ref }}
• deploy-aws-fargate.yml:28-30 -> deploy-fargate-${{ github.ref }}
• deploy-gcp.yml:27-29 -> deploy-gcp-${{ github.ref }}

Half 2, unconditional destruction of another writer's live lock, if: failure() || cancelled(), no age check, no held-lock check:

• deploy-gcp.yml:156-167 -> gsutil rm "gs://${BUCKET}/github-${ENVIRONMENT}/default.tflock"
• deploy-aws-lambda.yml:269-280 -> aws s3 rm "s3://${BUCKET}/github-${ENVIRONMENT}/terraform.tfstate.tflock"

Deleting another run's live lock object is exactly as destructive as breaking its lease. The lock mechanism differs; the failure mode does not. So the same silent state loss is reachable on GCP and AWS Lambda today.

Per the scope given for this change, that is flagged, not fixed. Widening it is a separate decision, and the AWS/GCP fix is not a copy-paste of this one (different backends, different lock semantics, deploy-aws-lambda.yml derives its environment differently). Filed as #1806.

Follow-ups filed

• Both halves of #1801 are still live on GCP and AWS: branch-keyed concurrency plus unconditional lock deletion #1806 (p1) BOTH halves of Azure deploy silently loses Terraform state: branch-keyed concurrency plus an unconditional lease break #1801 live on GCP and AWS, per the section above.
• #1807 (p2) nothing enforces the shared azure-tfstate-* group across the three writers; a fourth writer added without it silently does not serialize.
• Reusable deploy workflows read github.event_name as workflow_call, silently downgrading a release deploy to dev #1805 (p2) inside a reusable workflow github.event_name is the caller's event, so deploy-azure.yml's workflow_call case arm is dead code and a release-triggered deploy-all.yml silently deploys to dev while reporting prod. Affects deploy-azure.yml, deploy-gcp.yml and deploy-aws-fargate.yml; deploy-aws-lambda.yml:110-118 already handles it and is the reference implementation. Does not affect this PR: the group and the state key both read the same needs.prepare.outputs.environment, so whatever prepare decides, the group names the blob that actually gets written. It is a wrong-environment bug, not a serialization bug.

Two pre-existing defects also surfaced during review and are not filed, since they are adjacent and unverified by me end to end: rollback.yml:49 pins TF_VERSION: 1.6.0 against a stack requiring >= 1.10.0 (so rollback-azure may not init at all today), and database-migration.yml:431 runs a bare terraform init with no -backend-config against a partial backend block. Flagging rather than filing so someone with more context decides.

Summary by CodeRabbit

• Reliability
  • Improved Azure deployment, staging cleanup, and rollback coordination to prevent conflicting operations.
  • Preserved queued workflow runs instead of cancelling them when another operation is in progress.
  • Updated Terraform state handling to rely on standard locking and explicit recovery procedures.
• Documentation
  • Added workflow guidance explaining deployment concurrency behavior and Terraform state protection.

[coderabbitai]

📝 Walkthrough

Walkthrough

Azure workflows now serialize Terraform operations by environment state. Automatic Azure Blob lease breaking was removed. The deployment caller documents its deliberate lack of duplicate concurrency control.

Changes

Azure Terraform state locking

Layer / File(s)	Summary
Deployment state guard .github/workflows/deploy-azure.yml	The deployment job serializes runs by validated environment without cancellation. Workflow-level branch concurrency and automatic lease-breaking steps were removed.
Workflow state alignment .github/workflows/cleanup-staging.yml, .github/workflows/rollback.yml, .github/workflows/deploy-all.yml	Staging cleanup and rollback serialize Terraform operations by environment. The deployment caller documents why it does not add a second concurrency guard.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Possibly related issues

• Azure deploy silently loses Terraform state: branch-keyed concurrency plus an unconditional lease break #1801: Addresses branch-keyed concurrency and unconditional Azure lease breaking in Terraform workflows.
• LeanerCloud/CUDly#1807: Modifies the same Azure state-writer jobs and azure-tfstate-* concurrency groups.
• Both halves of #1801 are still live on GCP and AWS: branch-keyed concurrency plus unconditional lock deletion #1806: Applies state-keyed concurrency and lock-removal changes to Azure workflows.
• sec(ci): destroy-fargate-dev.yml force-deletes the state lock and matches ECR repos by substring #1592: Addresses unsafe Terraform state-lock handling and state-based concurrency.

Possibly related PRs

• LeanerCloud/CUDly#818: Both PRs remove automatic Terraform state-lock cleanup from CI workflows.
• LeanerCloud/CUDly#1131: Both PRs modify the Azure deployment workflows, including Terraform state-lock handling.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely summarizes the main change: serializing Azure Terraform state writers by environment instead of branch.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/1801-azure-tfstate-concurrency

---

_{Comment @coderabbitai help to get the list of available commands.}

[cristim]

Independent adversarial review — head ef451ad7ebc15ae687ead0efb475ae7939e41af6

Reviewed as a stranger's diff, verified against the files rather than the PR summary. All four workflow files were byte-identical at review time and at the committed SHA.

Status at time of posting: F1, F2 and F3 below already have fixes staged locally but not yet pushed, so they are still live against this PR head. Those fixes introduce two new comment-accuracy issues, recorded as F7 and F8 in the delta section at the end. F4, F5, F6 are unaddressed (F4/F5 are deliberately separate issues).

Verdict: the fix is sound. The concurrency group and the state key each derive from a single value in all three writers, so group and blob are in lockstep by construction — no writer can hold a group that does not name its own state file. One actionable item in the diff itself (F1); everything else is either a new failure mode worth documenting or a pre-existing adjacent defect for separate issues.

F1 — HIGH — .github/workflows/deploy-all.yml:167: the comment asserts an environment equality that does not hold on the release path

GitHub docs, verbatim:

When a reusable workflow is triggered by a caller workflow, the github context is always associated with the caller workflow.

So github.event_name is never the literal workflow_call, and the workflow_call arm at deploy-azure.yml:89 is dead code. This repo already documents the behaviour at deploy-aws-lambda.yml:110-115 and handles release) TARGET_ENVIRONMENT=prod ;; at line 117; deploy-azure.yml does not. On the release trigger, deploy-all.yml:60-64 sets environment=prod and passes it at line 180, but prepare falls into *) ENVIRONMENT=dev.

The new comment reads as though <environment> is deploy-all's environment. On a release, deploy-all reports prod while the group is azure-tfstate-dev.

Does this break the fix? No. deploy-azure.yml:124 (group) and :163 (state key) both read the one value needs.prepare.outputs.environment. Whatever prepare decides, the group names exactly the blob written. The same property holds cross-workflow: cleanup-staging.yml:260 vs :303, rollback.yml:407 vs :466. The mis-resolution changes which environment is deployed, never whether two writers of one blob serialize.

Fix here: reword the comment so it does not imply the two environments agree. Separate issue: the prepare bug at deploy-azure.yml:88-91, copying the pattern from deploy-aws-lambda.yml:110-118. The same latent bug is in deploy-gcp.yml:75-78 and deploy-aws-fargate.yml:83-86. No run-log evidence exists either way — gh run list --workflow deploy-all.yml returns [], that workflow has never run.

F2 — MEDIUM — .github/workflows/rollback.yml:406-413: an environment-approval gate may now hold the tfstate group

rollback-azure is bound to azure-${{ inputs.environment }}-rollback, and rollback.yml:19-22 says those environments must be given required reviewers. If a job waiting on approval holds its concurrency group, a rollback parked for a human blocks every deploy to that environment for the whole approval window.

Undocumented either way — neither the manage-environments nor the review-deployments page says whether a waiting job holds a group, and the concurrency reference does not cover the waiting state. Not provable without a live run. Inert today (cleanup-staging.yml:31-37 states no environment in this repo has protection rules), but it activates the moment rollback.yml:19-22 is acted on. Suggest flagging the unknown in the comment rather than leaving it to be discovered.

F3 — MEDIUM — all three sites: one pending slot per group, now shared across workflows

GitHub docs, verbatim:

When a concurrent job or workflow is queued, if another job or workflow using the same concurrency group in the repository is in progress, the queued job or workflow will be pending.

By default, any existing pending job or workflow in the same concurrency group will be canceled and the new queued job or workflow will take its place.

That eviction is the baseline and is independent of cancel-in-progress, which governs in-progress runs only. Before this change the groups were per-workflow, so eviction only ever dropped a redundant deploy. Now a queued rollback-azure on dev can be evicted by the next main push. Partial mitigation already present: rollback.yml:589-590 exits 1 on any non-success result, so a cancelled rollback reddens the run rather than vanishing silently. Worth one sentence in the rollback comment.

F4 — LOW, pre-existing, separate issue — rollback.yml:49 vs terraform/environments/azure/main.tf:5

TF_VERSION is pinned 1.6.0; the azure stack requires >= 1.10.0. rollback-azure's terraform init fails the version constraint before it can write anything, so rollback.yml:401-402's "terraform apply against github-<environment>.terraform.tfstate" is aspirational today. deploy-azure.yml:68 and cleanup-staging.yml:53 both use 1.10.0. Adding the group is still correct.

F5 — LOW, pre-existing, separate issue — database-migration.yml:431-432

Runs a bare terraform init with no -backend-config against the partial backend "azurerm" {} at terraform/environments/azure/backend.tf:2-4. There is no backend to initialize; this should fail in CI.

F6 — NIT — cleanup-staging.yml:282-287

After the lease-break deletion, destroy-azure makes no az calls at all. Terraform's OIDC path uses ARM_USE_OIDC plus the runner's ACTIONS_ID_TOKEN_*, not azure/login's session. Flagged only so nobody assumes the step is load-bearing — do not remove it blind.

---

Checked and clean

Empty / partial group key. Job-level concurrency evaluates after needs: prepare resolves; prepare exits 1 outside dev|staging|prod (deploy-azure.yml:96-102, under set -euo pipefail). On push, inputs is null so INPUT_ENVIRONMENT renders empty and the *) arm forces dev. Cannot collapse to a bare azure-tfstate-.

Deadlock. jobs.<job_id>.concurrency is a documented-permitted key on a job that calls a reusable workflow, and actionlint exits 0 on a variant adding it to deploy-all.yml's deploy-azure — so the comment describes a genuinely reachable misconfiguration, not a hypothetical. With cancel-in-progress: false the prediction holds: the caller holds the group, the called job goes pending on it, and the caller can never finish. GitHub documents the adjacent variant:

[!NOTE] If you use jobs.<job_id>.concurrency.cancel-in-progress: true, don't use the same value for jobs.<job_id>.concurrency.group in the called and caller workflows as this will cause the workflow that's already running to be cancelled.

Optional: mention that the docs call out the true case, so a future reader who flips the flag does not conclude the note stopped applying.

Completeness. rg -n 'environments/azure' .github/workflows/ Makefile* scripts/ yields exactly four sites. Writers — deploy-azure.yml:167/179/235 (init / plan / apply; terraform plan also takes the state lock and is inside the grouped job), cleanup-staging.yml:307/315, rollback.yml:467 — all grouped. Reader — database-migration.yml:431-433: terraform output has no -lock flag and does not lock, and init only touches state with -migrate-state, which is not passed. Correctly omitted. No Makefile target, no scripts/ invocation of the azure stack, no terraform -chdir in any workflow. deploy-all.yml:178 is the only caller of deploy-azure.yml.

Group-name consistency. Parsed via YAML rather than eyeballed. Prefix azure-tfstate- byte-identical at deploy-azure.yml:124, cleanup-staging.yml:260, rollback.yml:407; all suffixes lowercase dev|staging|prod. Docs confirm group names are case-insensitive, so there is no case hazard either way.

Broken references from the deletions. rg over .github/ runbooks/ docs/ scripts/ for STORAGE_ACCOUNT|lease break|Break stale|Release state lock|force-unlock finds no surviving consumer of the deleted variables. The deleted steps carried no id:. /tmp/backend.tfbackend is still written unconditionally before every azure terraform init: deploy-azure.yml:163→168, cleanup-staging.yml:303→308, rollback.yml:466→468, with no if: on any of them. The surviving lock-release steps at deploy-aws-lambda.yml:269 and deploy-gcp.yml:156 target S3/GCS on different states and are correctly untouched.

YAML / schema — validated in both directions. actionlint on all four files reports only pre-existing shellcheck SC2086/SC2129 info/style in untouched run: blocks; zero findings on any concurrency block. Because a clean result is only meaningful if the check is live, three positive controls were run on scratchpad copies: the same expression moved to workflow level gives context "needs" is not allowed here. available contexts are "github", "inputs", "vars"; steps. at job level gives available contexts are "github", "inputs", "matrix", "needs", "strategy", "vars"; a needs.prepar typo gives property "prepar" is not defined. All three match the docs context table exactly, and inputs in jobs.<id>.concurrency is explicitly in the allowed set.

Comments. Every verifiable claim holds. deploy-azure.yml:29-35's "limited to github, inputs, vars" is confirmed verbatim by both the docs table and the control above. :120-122 — cancel-in-progress does default to false, so "stated explicitly" is accurate. :152-153 — correct, and the mechanism is worth stating: the azurerm backend acquires an infinite lease, and az storage blob lease break with no --lease-break-period breaks an infinite lease immediately, whereas a fixed-duration lease would run out its remainder. :304-311 — on an ordinary failure Terraform does release the lock; on cancelled() GitHub sends SIGINT and then runs the if: cancelled() steps while Terraform is still shutting down, so the race described is real; a hard runner death runs no steps at all. Only deploy-all.yml:167 misleads (F1).

Not provable without a live run

1. F2 — whether the concurrency group is claimed before or after the environment-approval wait.
2. F3 — cross-workflow pending eviction is documented, not observed here.
3. The repro plan. deploy-azure.yml triggers on push: branches: [main] only, so the issue's June 9 evidence run on feat/multicloud-web-frontend (27217251153) cannot have come from a push under the current trigger config — it was either a workflow_dispatch on that ref or the branch filter differed at the time. Worth confirming which before writing "trigger two overlapping runs on different branches" into the verification steps. Either way the fix holds: two dispatches on different refs with the same environment now land in one group.

Sources: reusing-workflow-configurations, workflow-syntax#concurrency, contexts

---

Delta review of the staged (unpushed) fixes for F1-F3

F1, F2 and F3 are correctly addressed in substance. actionlint still exits 0 on all four files. Two accuracy problems in the new comment text:

F7 — MEDIUM — rollback.yml, new comment on rollback-azure: bakes a live-configuration claim into a comment, and misattributes it

The new text reads "if the environment: binding below is ever given required reviewers (see the header note; none are configured today)". Two problems:

1. Misattribution. rollback.yml's own header note (lines 19-22) does not say none are configured — it says the opposite in spirit, that these environments must be configured with required reviewers. The "no environment in this repo has any" claim lives in a different file, cleanup-staging.yml:32-34.
2. This is exactly the rot the sibling file refuses to create. cleanup-staging.yml:37-39 says, verbatim: "See sec(ci): deployment environments have no protection rules, so environment-bound credentialed jobs run unapproved cloud-commitments-platform#141 for the live state; deliberately not restated here so this comment cannot rot into false reassurance." The new rollback comment restates precisely that live state. The moment someone configures reviewers on azure-prod-rollback — which rollback.yml:19-22 instructs them to do — the comment becomes false, and it is false in the reassuring direction.

Fix: drop "none are configured today" and point at LeanerCloud/cloud-commitments-platform#141 for live state, matching cleanup-staging.yml:37-39. The conditional framing ("if ... is ever given required reviewers") already carries the meaning without asserting current config.

F8 — LOW — deploy-all.yml, new comment: the documented cancellation is one-directional, not mutual

The new text says GitHub "documents the sibling hazard for cancel-in-progress: true (caller and called sharing a group cancel each other)". The actual note is one-directional — it is the already-running caller that gets cancelled when the called workflow starts:

... don't use the same value for jobs.<job_id>.concurrency.group in the called and caller workflows as this will cause the workflow that's already running to be cancelled.

Fix: "cancels the already-running caller" rather than "cancel each other". Substance of the surrounding claim is unaffected.

Everything else in the two staged hunks checks out: the F1 rewording no longer asserts that deploy-all's environment and the inner group agree; the F3 wording matches the docs; and the "summary job below exits 1 on any non-success result" claim is verified at rollback.yml:600.

[cristim]

@coderabbitai review

The automatic review on this PR hit the adaptive rate limit before it ran, so no review has landed yet. Requesting it now that the window has reopened. Head is 2c4f88ec6; the only functional commit is ef451ad7e, the three after it are comment-only.

[coderabbitai]

@cristim I requested a full review because the earlier automatic review was rate-limited. This review covers head 2c4f88ec6, including the functional commit and the three comment-only commits.

---

🧠 Learnings used

Learnt from: cristim
Repo: LeanerCloud/CUDly PR: 0
File: :0-0
Timestamp: 2026-06-26T17:23:32.317Z
Learning: In this repository, if a CodeRabbit review was throttled or hit a rate limit, the correct recovery is to request `coderabbitai full review` rather than `coderabbitai review`, because incremental review can silently skip the affected in-flight commit and report a false-clean result.

---

Learnt from: cristim
Repo: LeanerCloud/CUDly PR: 0
File: :0-0
Timestamp: 2026-06-26T17:21:01.385Z
Learning: For the LeanerCloud/CUDly repository review workflow, when a previous CodeRabbit review pass was skipped or failed to produce findings due to a rate-limit event, use a full review request on the pull request rather than the incremental review form.

---

Learnt from: cristim
Repo: LeanerCloud/CUDly PR: 0
File: :0-0
Timestamp: 2026-08-06T08:28:24.968Z
Learning: In the LeanerCloud/CUDly repository, request CodeRabbit reviews at most once per hour across the repository because the adaptive quota is shared across open pull requests. When a PR head was pushed while automatic review was quota-exhausted and was not retried, use a full review rather than an incremental review so the missed commits are reviewed.

---

Learnt from: cristim
Repo: LeanerCloud/CUDly PR: 0
File: :0-0
Timestamp: 2026-08-05T05:27:02.254Z
Learning: For the LeanerCloud/CUDly repository, pace CodeRabbit review requests at one request per hour across the repository. The review quota is adaptive and shared per developer and organization; burst requests can exhaust the quota and tighten the limit. When a PR head was pushed while quota was exhausted, request a full review because an incremental review skips the unreviewed commits.

---

Learnt from: cristim
Repo: LeanerCloud/CUDly PR: 0
File: :0-0
Timestamp: 2026-08-04T08:23:48.546Z
Learning: For the LeanerCloud/CUDly repository, CodeRabbit review quota is shared per developer across the organization and adapts to recent review volume. Space review requests to approximately one per hour. When an automatic review did not run because quota was exhausted, use a full review instead of an incremental review so CodeRabbit inspects the missed commits.

---

Learnt from: cristim
Repo: LeanerCloud/CUDly PR: 0
File: :0-0
Timestamp: 2026-08-04T22:26:24.782Z
Learning: For pull request `#1716`, when review quota exhaustion prevents automatic review of pushed commits, request a full review instead of an incremental review. An incremental review can omit those unreviewed commits.

⚠️ Action not completed

Review rate limited.

---

Your included review limit is currently reached under our Fair Usage Limits Policy. Your recent PR review activity is in the 95th percentile or higher among CodeRabbit users, so adaptive limits apply. This review may still proceed through usage-based billing if eligible. Your next included review will be available in 11 minutes.

[cristim]

@coderabbitai full review

The automatic review on this PR hit the adaptive rate limit and never ran, so no review has landed. Requesting a full review rather than incremental, since an incremental review after a throttle can skip the commits that were in flight.

Head is 2c4f88ec6. The only functional commit is ef451ad7e; the three after it are comment-only.

Worth your attention: the change moves Terraform state serialization from a branch-keyed workflow-level concurrency to an environment-keyed job-level group, and deletes three az storage blob lease break steps. The load-bearing question is whether the group key can ever evaluate empty or diverge from the state-file key it is meant to protect, since either would silently reintroduce the concurrent-writer bug this fixes.

[coderabbitai]

@cristim I requested a full review for pull request #1803 at head 2c4f88ec6.

This uses the repository guidance for rate-limited reviews: a full review includes commits that an incremental review can omit.

---

🧠 Learnings used

Learnt from: cristim
Repo: LeanerCloud/CUDly PR: 0
File: :0-0
Timestamp: 2026-08-03T16:30:31.562Z
Learning: In `.github/workflows/deploy-aws-lambda.yml`, the `prepare` job intentionally maps `release` events to `prod`, `workflow_dispatch` events to the requested environment, and all other events, including pushes to `main`, to `dev`. The fallback prevents unrecognized triggers from selecting a production-adjacent environment. `.github/workflows/README.md` must document pushes to `main` as deployments to `dev`.

---

Learnt from: cristim
Repo: LeanerCloud/CUDly PR: 1657
File: .github/workflows/deploy-aws-lambda.yml:110-120
Timestamp: 2026-08-03T16:30:01.796Z
Learning: In `.github/workflows/deploy-aws-lambda.yml`, the `prepare` job intentionally maps unrecognised `github.event_name` values, including `push`, to the `dev` target environment. Do not change this fallback to `staging`; documentation must describe push deployments as deployments to `dev`.

---

Learnt from: cristim
Repo: LeanerCloud/CUDly PR: 0
File: :0-0
Timestamp: 2026-06-26T17:23:32.317Z
Learning: In this repository, if a CodeRabbit review was throttled or hit a rate limit, the correct recovery is to request `coderabbitai full review` rather than `coderabbitai review`, because incremental review can silently skip the affected in-flight commit and report a false-clean result.

---

Learnt from: cristim
Repo: LeanerCloud/CUDly PR: 0
File: :0-0
Timestamp: 2026-06-26T17:21:01.385Z
Learning: For the LeanerCloud/CUDly repository review workflow, when a previous CodeRabbit review pass was skipped or failed to produce findings due to a rate-limit event, use a full review request on the pull request rather than the incremental review form.

---

Learnt from: cristim
Repo: LeanerCloud/CUDly PR: 0
File: :0-0
Timestamp: 2026-08-06T08:28:24.968Z
Learning: In the LeanerCloud/CUDly repository, request CodeRabbit reviews at most once per hour across the repository because the adaptive quota is shared across open pull requests. When a PR head was pushed while automatic review was quota-exhausted and was not retried, use a full review rather than an incremental review so the missed commits are reviewed.

---

Learnt from: cristim
Repo: LeanerCloud/CUDly PR: 0
File: :0-0
Timestamp: 2026-08-05T05:27:02.254Z
Learning: For the LeanerCloud/CUDly repository, pace CodeRabbit review requests at one request per hour across the repository. The review quota is adaptive and shared per developer and organization; burst requests can exhaust the quota and tighten the limit. When a PR head was pushed while quota was exhausted, request a full review because an incremental review skips the unreviewed commits.

---

Learnt from: cristim
Repo: LeanerCloud/CUDly PR: 0
File: :0-0
Timestamp: 2026-08-04T08:23:48.546Z
Learning: For the LeanerCloud/CUDly repository, CodeRabbit review quota is shared per developer across the organization and adapts to recent review volume. Space review requests to approximately one per hour. When an automatic review did not run because quota was exhausted, use a full review instead of an incremental review so CodeRabbit inspects the missed commits.

---

Learnt from: cristim
Repo: LeanerCloud/CUDly PR: 0
File: :0-0
Timestamp: 2026-08-04T22:26:24.782Z
Learning: For pull request `#1716`, when review quota exhaustion prevents automatic review of pushed commits, request a full review instead of an incremental review. An incremental review can omit those unreviewed commits.

✅ Action performed

Full review finished.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/deploy-all.yml:
- Around line 166-170: Update the concurrency explanation comment in the
workflow_call section to state that inputs.environment is passed through
directly to the called workflow and matches the caller’s computed environment;
remove the claim that the called workflow’s prepare output may differ from
environment.

🪄 Autofix

❌ Autofix failed (check again to retry)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 6be1c247-7c26-42da-b0d6-d568ae4c06e6

📥 Commits

Reviewing files that changed from the base of the PR and between 51e89a6 and 2c4f88e.

📒 Files selected for processing (4)

• .github/workflows/cleanup-staging.yml
• .github/workflows/deploy-all.yml
• .github/workflows/deploy-azure.yml
• .github/workflows/rollback.yml

[coderabbitai]

Autofix skipped. No unresolved CodeRabbit review comments with fix instructions found.