Repository navigation
Commit 2eff8a4
authored
fix(ci): key rollback-aws-fargate on the Fargate Terraform state (#1857)
Fargate rollback wrote the Lambda Terraform state
rollback.yml's rollback-aws-fargate job built its backend key from the
Lambda state namespace, writing github-<env>/terraform.tfstate where every
other Fargate writer uses github-fargate-<env>/. A Fargate rollback
therefore initialised against the state the Lambda deploy owns and applied
compute_platform=fargate into it: against a populated Lambda state that is
a platform swap of the live Lambda stack recorded in the wrong file, while
the real Fargate state is left describing resources nobody reconciles. The
Lambda rollback job uses the same key correctly, so both rollback jobs were
writing one state file.
The key now matches the platform the job applies. Enumerated rather than
sampled: all 17 workflow files were swept for backend keys, compute_platform
vars and concurrency groups, and every writer cross-read in both directions.
Exactly one defective site, not two. The adjacent sites that are not
instances are recorded in the PR: the Azure blob has no platform split, GCP
uses prefix rather than key, and init-backend.sh emits an unprefixed key.
Blast radius, checked separately because a fix does not answer it:
rollback.yml has never run. The empty run list was validated in the other
direction by running the same query against ci.yml and getting rows back,
so it means no runs rather than a wrong filter. The corruption is latent,
nothing needs terraform state mv, and repointing the key strands nothing.
Not closed: reading the state object in S3 needs credentials unavailable
here, and a hand-run apply from a laptop would leave no trace in the repo.
A guard makes the pairing durable, since reading carefully once does not
survive the next edit. It sweeps the workflow directory and every script,
asserts the swept set is non-empty and contains the known site, and fails
if a job's backend key disagrees with the platform it applies.
Review found two ways that guard failed open, both fixed.
A job with the correct platform and key but no concurrency group at all
passed, because the group was only checked when non-empty. Absence was
read as an exemption rather than as a missing requirement, and such a job
can apply shared state with no workflow-level serialization. It is now two
independent conditions rather than an if/else on a non-empty group.
build_swept_scripts globbed only scripts/*.sh and scripts/lib/*.sh, so a
file at scripts/aws/rollback.sh was never opened while all three suites
reported coverage of every script under scripts/. Fixed in the shared
helper rather than locally, because the RDS and ECR guards call the same
function: mutation confirmed both of those suites, as merged on main, pass
with a nested offender present, so they have carried this blind spot since
#1852. Discovery uses find -print0 with read -d '' rather than a ** glob,
since globstar is bash 4 and these scripts run on the bash 3.2 that ships
with macOS, with sort -z for deterministic order and process substitution
so the array survives the loop.
Verified: mutations run against copies of the tree, each required to
produce its specific FAIL line since a syntax error also exits non-zero.
With a nested offender present all three suites now fail naming it, and
with it removed all three go green. ECR 48/48 and RDS 53/53 still pass
against the recursive helper.
Stated rather than glossed: scripts/ contains no nested *.sh today, so
recursive and non-recursive discovery return a byte-identical 26 files.
The recursion fix is preventive, and the recursion assertion runs against
a fixture tree, because asserting recursion where no nested file exists
would pass without testing anything.
Not verified: no terraform apply, init against a real backend, or destroy
was run, and neither destroy workflow was dispatched.
Deferred: #1856, database-migration.yml runs a bare terraform init with no
-backend-config in three jobs and swallows the real terraform error before
reading terraform output.1 parent 30f700a commit 2eff8a4
1 file changed
Lines changed: 35 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
824 | 824 | | |
825 | 825 | | |
826 | 826 | | |
| 827 | + | |
| 828 | + | |
| 829 | + | |
| 830 | + | |
| 831 | + | |
| 832 | + | |
| 833 | + | |
| 834 | + | |
| 835 | + | |
| 836 | + | |
| 837 | + | |
| 838 | + | |
| 839 | + | |
| 840 | + | |
| 841 | + | |
| 842 | + | |
| 843 | + | |
| 844 | + | |
| 845 | + | |
| 846 | + | |
| 847 | + | |
| 848 | + | |
| 849 | + | |
| 850 | + | |
| 851 | + | |
| 852 | + | |
| 853 | + | |
| 854 | + | |
| 855 | + | |
| 856 | + | |
| 857 | + | |
| 858 | + | |
| 859 | + | |
| 860 | + | |
827 | 861 | | |
828 | 862 | | |
829 | 863 | | |
| |||
863 | 897 | | |
864 | 898 | | |
865 | 899 | | |
| 900 | + | |
866 | 901 | | |
867 | 902 | | |
868 | 903 | | |
| |||
0 commit comments