Skip to content

ci: add per-database concurrency groups to database-migration migrate jobs - #1816

Merged
cristim merged 1 commit into
mainfrom
fix/1593-database-migration-concurrency
Aug 13, 2026
Merged

cristim merged 1 commit into
mainfrom
fix/1593-database-migration-concurrency

Conversation

@cristim

@cristim cristim commented Aug 13, 2026 •

Copy link
Copy Markdown
Member

Closes #1593

Why the diff is smaller than the issue title

#1593 names four workflows. Three of them are already covered on main:
rollback.yml and cleanup-staging.yml by #1806/#1801, and
destroy-fargate-dev.yml by #1806. Enumerating every job on origin/main
(23ad4e4) confirms it, and shows database-migration.yml as the only one
with no group at the workflow level or on any of its five jobs:

workflow top-level jobs with a group
rollback.yml none 4/6 (rollback-*; validate/summary exempt)
cleanup-staging.yml none 4/5 (destroy-*; guard exempt)
destroy-fargate-dev.yml none 1/2 (destroy; guard exempt)
database-migration.yml none 0/5

So this PR only touches the fourth.

The change

Job-level concurrency on migrate-aws, migrate-gcp and migrate-azure.

Why not the *-tfstate-* groups. These jobs are not Terraform state
writers. They run terraform init plus terraform output -raw to read a
database endpoint, which takes no state lock. Reusing
aws-tfstate-*/gcp-tfstate-*/azure-tfstate-* would serialize migrations
against unrelated deploys and still not identify the database. The group keys
on the resource that actually needs protecting, db-migrate-<cloud>-<env>:

  • two dispatches for the same cloud and environment serialize;
  • different clouds, and the same cloud in different environments, stay
    concurrent, so cloud=all still fans out across the three clouds.

cancel-in-progress: false. Cancelling mid-migrate leaves
schema_migrations dirty, which then needs the CUDLY_FORCE_MIGRATION_VERSION
recovery path in internal/database/postgres/migrations/migrate.go.

Empty-group risk, handled explicitly. environment is a required choice
constrained to dev|staging|prod on workflow_dispatch, but on the
workflow_call trigger it is a plain type: string; required: true makes a
caller pass the key, but GitHub does not enforce a non-empty value. An empty
interpolation would collapse every environment into one group, so the
expression falls back to an unset sentinel. validate rejects an environment
outside the allowlist before any migrate-* job runs, so the sentinel should be
unreachable today; it is there so the group cannot become silently ambiguous if
that ordering changes. There is currently no in-repo uses: caller of this
workflow.

validate and summary touch no database and get no group. No top-level group
either, since that would serialize the three clouds against each other.

Verification

  • YAML parses before and after (yaml.safe_load).
  • Post-change enumeration over the defining predicate, not a sample: 3/3
    migrate-* jobs have a group with cancel-in-progress: false, 0 uncovered,
    3 distinct groups, 0 non-migrate-* jobs given a group, top-level still
    None.
  • actionlint reports the identical 5 pre-existing shellcheck info findings
    before and after (line numbers shift only); no new findings. Confirmed
    non-vacuous by a negative control: swapping in a bogus inputs.* reference
    makes actionlint flag that exact group: line, so it does type-check the
    key. Note actionlint is not wired into this repo's CI or pre-commit; this
    was a local run.
  • Pre-commit hooks passed; nothing bypassed.

Runtime concurrency behaviour cannot be verified without dispatching real
migrations, so it is argued from the trigger definitions and GitHub's
documented semantics rather than observed.

Noted, not fixed here

Adding a group makes eviction possible: GitHub keeps one pending entry per
group, so a queued migration can be cancelled by a later dispatch. Unlike
rollback.yml, this workflow's summary job does not exit 1 on a non-success
result, so an evicted migration would leave the run green. That is a separate
concern from #1593 and is documented inline rather than changed here. LeanerCloud/cloud-commitments-platform#114
(terraform init with no -backend-config) is likewise out of scope.

Summary by CodeRabbit

  • Reliability
    • Improved database migration scheduling across AWS, GCP, and Azure.
    • Migrations are now serialized by cloud and environment while allowing work for different databases to proceed independently.
    • In-progress migrations are preserved rather than canceled when new runs start.

… jobs

The three `migrate-*` jobs had no concurrency group, so two dispatches for
the same cloud and environment could run `migrate` concurrently against one
schema_migrations table.

Each job now takes a `db-migrate-<cloud>-<environment>` group. These jobs are
not Terraform state writers: they run `terraform init` plus `terraform output
-raw` to read a database endpoint and take no state lock, so reusing the
`aws-tfstate-*` / `gcp-tfstate-*` / `azure-tfstate-*` groups from #1806 and
#1801 would serialize migrations against unrelated deploys while still not
identifying the database. The group keys on the resource that actually needs
protecting instead, leaving different clouds and different environments free
to run concurrently, including under `cloud=all`.

`cancel-in-progress: false` throughout: cancelling mid-`migrate` leaves
schema_migrations dirty and needs the CUDLY_FORCE_MIGRATION_VERSION recovery
path.

`inputs.environment` is a required `choice` on `workflow_dispatch` but a plain
`type: string` on `workflow_call`, where GitHub does not enforce a non-empty
value, so the group expression falls back to an `unset` sentinel rather than
collapsing every environment into one group on an empty interpolation.

`validate` and `summary` touch no database and are deliberately left without a
group, as is the workflow top level, which would otherwise serialize the three
clouds against each other.

Refs #1593
@cristim cristim added priority/p1 Next up; this sprint severity/high Significant harm urgency/this-sprint Within the current sprint impact/internal Team-internal only effort/s Hours type/bug Defect triaged Item has been triaged labels Aug 13, 2026
@coderabbitai

coderabbitai Bot commented Aug 13, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: f03e65dc-5ac1-4f68-8b9f-e95d145c8ee4

📥 Commits

Reviewing files that changed from the base of the PR and between 23ad4e4 and 9fadbbb.

📒 Files selected for processing (1)
  • .github/workflows/database-migration.yml

📝 Walkthrough

Walkthrough

The database migration workflow adds separate concurrency controls for AWS, GCP, and Azure. Each control scopes migration runs by cloud and environment, preserves parallelism across clouds, and prevents cancellation of active migrations.

Changes

Migration concurrency

Layer / File(s) Summary
Cloud and environment concurrency groups
.github/workflows/database-migration.yml
AWS, GCP, and Azure migration jobs now use cloud- and environment-scoped concurrency groups. Empty workflow-call environments fall back to unset, and active jobs are not cancelled.

Estimated code review effort: 2 (Simple) | ~10 minutes

Mergeability Score: ⚪ Minimal · up to 9fadb

This change adds per-database concurrency to migration jobs while preserving safe parallelism across clouds and environments; no actionable merge-blocking risk remains after normal checks and review.

Possibly related PRs

  • LeanerCloud/CUDly#1803: Adds similar cloud/environment-scoped concurrency controls to Azure Terraform state-writing workflows.
  • LeanerCloud/CUDly#1812: Adds similar non-canceling concurrency groups for other cloud workflows.
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly describes the main change: adding per-database concurrency groups to database migration jobs.
Linked Issues check ✅ Passed The PR adds per-cloud and per-environment, non-cancelling concurrency groups to all migration jobs required for the database-migration scope of issue #1593.
Out of Scope Changes check ✅ Passed All changes are limited to database migration concurrency controls and directly support issue #1593.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/1593-database-migration-concurrency

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

effort/s Hours impact/internal Team-internal only priority/p1 Next up; this sprint severity/high Significant harm triaged Item has been triaged type/bug Defect urgency/this-sprint Within the current sprint

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(ci): no concurrency group on rollback, database-migration, cleanup-staging or destroy-fargate-dev

1 participant