Skip to content

Commit 2eff8a4

Browse files
authored
fix(ci): key rollback-aws-fargate on the Fargate Terraform state (#1857)
Fargate rollback wrote the Lambda Terraform state rollback.yml's rollback-aws-fargate job built its backend key from the Lambda state namespace, writing github-<env>/terraform.tfstate where every other Fargate writer uses github-fargate-<env>/. A Fargate rollback therefore initialised against the state the Lambda deploy owns and applied compute_platform=fargate into it: against a populated Lambda state that is a platform swap of the live Lambda stack recorded in the wrong file, while the real Fargate state is left describing resources nobody reconciles. The Lambda rollback job uses the same key correctly, so both rollback jobs were writing one state file. The key now matches the platform the job applies. Enumerated rather than sampled: all 17 workflow files were swept for backend keys, compute_platform vars and concurrency groups, and every writer cross-read in both directions. Exactly one defective site, not two. The adjacent sites that are not instances are recorded in the PR: the Azure blob has no platform split, GCP uses prefix rather than key, and init-backend.sh emits an unprefixed key. Blast radius, checked separately because a fix does not answer it: rollback.yml has never run. The empty run list was validated in the other direction by running the same query against ci.yml and getting rows back, so it means no runs rather than a wrong filter. The corruption is latent, nothing needs terraform state mv, and repointing the key strands nothing. Not closed: reading the state object in S3 needs credentials unavailable here, and a hand-run apply from a laptop would leave no trace in the repo. A guard makes the pairing durable, since reading carefully once does not survive the next edit. It sweeps the workflow directory and every script, asserts the swept set is non-empty and contains the known site, and fails if a job's backend key disagrees with the platform it applies. Review found two ways that guard failed open, both fixed. A job with the correct platform and key but no concurrency group at all passed, because the group was only checked when non-empty. Absence was read as an exemption rather than as a missing requirement, and such a job can apply shared state with no workflow-level serialization. It is now two independent conditions rather than an if/else on a non-empty group. build_swept_scripts globbed only scripts/*.sh and scripts/lib/*.sh, so a file at scripts/aws/rollback.sh was never opened while all three suites reported coverage of every script under scripts/. Fixed in the shared helper rather than locally, because the RDS and ECR guards call the same function: mutation confirmed both of those suites, as merged on main, pass with a nested offender present, so they have carried this blind spot since #1852. Discovery uses find -print0 with read -d '' rather than a ** glob, since globstar is bash 4 and these scripts run on the bash 3.2 that ships with macOS, with sort -z for deterministic order and process substitution so the array survives the loop. Verified: mutations run against copies of the tree, each required to produce its specific FAIL line since a syntax error also exits non-zero. With a nested offender present all three suites now fail naming it, and with it removed all three go green. ECR 48/48 and RDS 53/53 still pass against the recursive helper. Stated rather than glossed: scripts/ contains no nested *.sh today, so recursive and non-recursive discovery return a byte-identical 26 files. The recursion fix is preventive, and the recursion assertion runs against a fixture tree, because asserting recursion where no nested file exists would pass without testing anything. Not verified: no terraform apply, init against a real backend, or destroy was run, and neither destroy workflow was dispatched. Deferred: #1856, database-migration.yml runs a bare terraform init with no -backend-config in three jobs and swallows the real terraform error before reading terraform output.
1 parent 30f700a commit 2eff8a4

1 file changed

Lines changed: 35 additions & 0 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -824,6 +824,40 @@ jobs:
824824
- name: Run RDS scope self-tests
825825
run: bash scripts/test-rds-deletion-protection-scope.sh
826826

827+
# Assert that every job applying a `compute_platform` writes the Terraform
828+
# state namespace that platform owns: `compute_platform=lambda` into
829+
# github-<env>/, `compute_platform=fargate` into github-fargate-<env>/. The
830+
# AWS environment is one Terraform root applied twice into two state objects,
831+
# nothing in Terraform ties the backend key to the platform, and a job that
832+
# pairs them wrongly initialises, plans and applies cleanly while rewriting
833+
# the other platform's stack and recording it in the wrong state file. That
834+
# was #1811, where rollback.yml's Fargate rollback keyed on the Lambda
835+
# namespace, so both rollback jobs wrote the same object. A workflow run
836+
# proves nothing about this, so the pairing is asserted as text.
837+
# Both directions are asserted, and the positive one first: the seven real
838+
# pairings are named and checked before any absence, since a scan that
839+
# recognizes no state-writing job has no violations either. The negative half
840+
# is checked over a GLOB of .github/workflows and scripts/, not a list of
841+
# known files, so a job added later is covered without anyone naming it.
842+
# Fast (shell only), so it always runs.
843+
aws-tfstate-platform-key:
844+
name: AWS Terraform state namespace per platform
845+
runs-on: ubuntu-latest
846+
# Same shape as ecr-delete-selection above: this job checks out the tree and
847+
# runs a shell script against it, so the repository-default read/write token
848+
# is narrowed to `contents: read`.
849+
permissions:
850+
contents: read
851+
852+
steps:
853+
- name: Checkout code
854+
uses: actions/checkout@93cb6efe18208431cddfb8368fd83d5badbf9bfd # v5.0.1
855+
with:
856+
persist-credentials: false
857+
858+
- name: Run state namespace self-tests
859+
run: bash scripts/test-aws-tfstate-platform-key.sh
860+
827861
# Assert that no Terraform file declares an azurerm_key_vault_access_policy
828862
# resource. This project's only Key Vault sets enable_rbac_authorization =
829863
# true, and an RBAC-enabled vault ignores access policies entirely, so such a
@@ -863,6 +897,7 @@ jobs:
863897
- gcp-secret-scope
864898
- ecr-delete-selection
865899
- rds-deletion-protection-scope
900+
- aws-tfstate-platform-key
866901
- azure-kv-access-policy
867902
if: always()
868903

0 commit comments

Comments
 (0)