Skip to content

fix(iac/aws): rollback environments are absent from the OIDC trust allowlist, so AWS/Azure rollback cannot authenticate #139

Description

@cristim

Found while fixing LeanerCloud/cloud-commitments-cli#1542 (PR LeanerCloud/cloud-commitments-cli#1641). This is a pre-existing broken deploy path, not a security hole, and not a regression introduced by LeanerCloud/cloud-commitments-cli#1641.

What

.github/workflows/rollback.yml binds its four rollback jobs to environments named:

aws-lambda-<env>-rollback
aws-fargate-<env>-rollback
gcp-<env>-rollback
azure-<env>-rollback

A job with an environment: binding gets the OIDC subject repo:<org/repo>:environment:<name>. But the AWS trust policy allowlist at terraform/environments/aws/ci-cd-permissions/role.tf:28-33 is exactly:

"repo:${var.github_repo}:ref:refs/heads/main",
"repo:${var.github_repo}:environment:dev",
"repo:${var.github_repo}:environment:staging",
"repo:${var.github_repo}:environment:prod",

environment:aws-lambda-prod-rollback is not in that list, so configure-aws-credentials in the rollback job fails with Not authorized to perform sts:AssumeRoleWithWebIdentity. The Azure federated credentials have the same shape (only ref:refs/heads/main and pull_request), so azure/login fails there too.

GCP is unaffected — terraform/environments/gcp/ci-cd-permissions/github_oidc.tf conditions on assertion.repository / assertion.ref and binds by attribute.repository, so the environment name is irrelevant.

Why it is pre-existing

The environment bindings predate LeanerCloud/cloud-commitments-cli#1641 — git show 3e9660d06:.github/workflows/rollback.yml shows rollback-aws-lambda already carrying environment: name: aws-lambda-${{ inputs.environment }}-rollback. So the terraform apply job could never authenticate, before or after.

What LeanerCloud/cloud-commitments-cli#1641 changes is when the operator finds out. Previously the ungated verify-image job authenticated successfully (its subject was ref:refs/heads/main, which the policy does accept) and the failure surfaced at the apply step. LeanerCloud/cloud-commitments-cli#1641 deletes verify-image — that ungated-but-credentialed job was the vulnerability — so the failure now surfaces after the approval prompt instead of before it. Both fail; neither ever completed a rollback.

Fix options

  1. Add the rollback environments to the allowlist — extend role.tf's sub list with repo:<org/repo>:environment:{aws-lambda,aws-fargate,azure}-{dev,staging,prod}-rollback and add matching azuread_application_federated_identity_credential subjects. Keeps a distinct reviewer gate for rollbacks, at the cost of 12 more subjects and 12 more environments to provision.
  2. Reuse the existing environments — rename the workflow's bindings to plain dev / staging / prod. One-line change, no IaC edit, but rollbacks then share the deploy environments' reviewer set rather than having their own.

Option 2 is the smaller change; option 1 is right if rollback is meant to have a different approver list than a normal deploy. That is a policy call, not a technical one.

Also worth noting

GitHub auto-creates a referenced environment on first use with no protection rules. Nothing in this repo provisions these environments or their required reviewers (no github_repository_environment resource anywhere under terraform/ or iac/). So even once the trust policy accepts the subject, the "reviewer gate" is only real if someone configures it in repo settings. Whichever option is chosen should include provisioning the environments and their reviewers, ideally in IaC so it is reviewable.

Note terraform/environments/*/ci-cd-permissions/ is bootstrap-only per CLAUDE.md — applied manually by a privileged human, not by the deploy workflow.

Related: LeanerCloud/cloud-commitments-cli#1591 (environment bindings on destructive workflows), #90 (Azure federated credential trusts pull_request), LeanerCloud/cloud-commitments-cli#1542 / LeanerCloud/cloud-commitments-cli#1641.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions