Skip to content

fix(iac/aws): unblock deploy on rds:DescribeDBInstances and close audited grant gaps - #1699

Merged
cristim merged 3 commits into
mainfrom
fix/deploy-role-rds-and-audit
Aug 3, 2026
Merged

cristim merged 3 commits into
mainfrom
fix/deploy-role-rds-and-audit

Conversation

@cristim

@cristim cristim commented Aug 3, 2026 •

Copy link
Copy Markdown
Member

Closes #1698 · follow-ups tracked in LeanerCloud/cloud-commitments-platform#157 and LeanerCloud/cloud-commitments-platform#160

The immediate failure

Deploy to AWS Lambda on main (run 30809235060) fails at Terraform Plan:

Error: reading RDS DB Instance (cudly-dev-426fc8af-postgres): operation error RDS: DescribeDBInstances,
StatusCode: 403, AccessDenied: ... is not authorized to perform: rds:DescribeDBInstances
on resource: arn:aws:rds:us-east-1:909626172446:db:*

rds:DescribeDBInstances was already granted, scoped to arn:aws:rds:*:*:db:cudly-*, and the instance name matches that prefix. The denial nevertheless names db:*.

Root cause. The Terraform id of aws_db_instance is the DbiResourceId (db-<opaque>), not the DB identifier. In terraform-provider-aws v5.100.0, findDBInstanceByID (internal/service/rds/instance.go) branches on the shape of the id and, for a DbiResourceId, sends DescribeDBInstances with Filters=[dbi-resource-id] and DBInstanceIdentifier nil. RDS authorizes an identifier-less DescribeDBInstances against the wildcard ARN db:*, which no name-scoped grant can match.

This is unconditional on every read, not a not-found fallback: the provider's retry with the plain identifier only fires on NotFound, and AccessDenied is not NotFound, so the plan hard-fails.

On the deposed objects. The repeated # (left over from a partially-failed replacement of this instance) lines in the plan output are deposed objects from earlier failed applies. They are a consequence of this, not the trigger, and are not chased here.

Why this is not another one-action patch

This grant has been patched four times by adding whatever 403'd next (#1496, #1514, #1671). Each fix was correct and each left the next gap in place. So this PR enumerates every resource/data block under terraform/environments/aws/ and its modules, lists the API calls provider v5.100.0 makes for read/create/update/delete, and diffs that against the union of the four policies. Deploy-path status was resolved against github-{dev,staging,prod}.tfvars and both deploy workflows.

The audit named a third failure mode the previous patches did not account for:

  1. Missing action — absent from the policy. Discoverable by eye.
  2. Granted but unsatisfiable condition — gated on a condition that cannot hold at call time (the fix(iac/aws): grant kms:TagResource for KMS CreateKey tag-on-create #1671 kms:TagResource bug; the fix(iac/aws): case-insensitive tag match on kms:GetKeyPolicy deploy grant #1514 tag-case bug).
  3. Granted but unmatchable resource — the ARN pattern can never match the request, either because the call carries no identifier, or because the real ARN uses a different segment or a server-assigned opaque id.

Classes 2 and 3 both look completely fine when you read the policy. All three classes turned up again in this audit, in code nobody had looked at.

Fixed here

Blocking the deploy path

Gap Class Effect if unfixed
rds:DescribeDBInstances scoped to db:cudly-* unmatchable resource plan fails today, all envs
ec2:CreateLaunchTemplateVersion, ec2:ModifyLaunchTemplate, ec2:DeleteLaunchTemplateVersions missing fck-nat's AMI comes from a most_recent = true lookup, so every republish takes the launch template's update path. Only Create/Delete were granted, so the first AMI rotation after the ASG exists fails the apply. All envs.
EC2RunInstancesFckNAT used StringEquals on ec2:InstanceType unsatisfiable condition AWS authorizes one RunInstances against every resource type it touches, and ec2:InstanceType is only in context for the instance leg. On the volume / ENI / security-group / subnet / image / launch-template legs the key is absent, StringEquals evaluates false, and the whole call is denied. Now StringEqualsIfExists, which keeps the t4g.nano restriction exactly where the key exists.
iam:PassRole missing ec2.amazonaws.com unsatisfiable condition CreateAutoScalingGroup dry-run validates PassRole for the fck-nat instance profile (enable_nat_gateway hardcoded true, networking.tf:16, all envs)
iam:PassRole missing vpc-flow-logs.amazonaws.com unsatisfiable condition apply fails creating/replacing aws_flow_log (staging + prod)
iam:PassRole missing events.amazonaws.com unsatisfiable condition apply fails on events:PutTargets with role_arn (Fargate path, enable_scheduled_tasks = true in all three tfvars)
iam:CreateServiceLinkedRole missing autoscaling, elasticloadbalancing, ecs missing SLRs are created lazily on first use of a service in an account/region: invisible where the role exists, hard failure in a fresh region. Note ecs.application-autoscaling was granted and is a different principal that does not cover plain ecs.
rds:DescribeDBSubnetGroups scoped to subgrp:cudly-* unmatchable resource same call shape as the DescribeDBInstances read that is 403ing. aws_db_subnet_group.main (terraform/modules/database/aws/main.tf:73) is created in every environment, so leaving it scoped primes the next 403.
rds:ModifyDBSubnetGroup missing apply fails if az_count or the private subnet set changes

Latent, fixed while the files were open

Gap Class Reachable when
CloudFrontMutateTaggedOnly gated on case-sensitive StringEquals "CUDly" unsatisfiable condition The distribution's Project tag resolves to lowercase cudly via local.common_tags, so all four of its actions were silently denied. This is #1496's KMS bug, unfixed on the CloudFront statement because enable_cdn = false meant nothing exercised it. It would also have killed the destroy path, since deleting a distribution requires UpdateDistribution to disable it first.
ec2:DeleteTags missing the other half of ec2:CreateTags; dropping any key from common_tags/default_tags on an existing EC2 resource
ec2:ReplaceRouteTableAssociation missing the association's update path calls it instead of Disassociate + Associate
ecr:PutImageScanningConfiguration missing any scan_on_push change; its sibling PutImageTagMutability was already granted
elasticloadbalancing:SetSubnets, SetSecurityGroups, SetIpAddressType, ModifyListenerAttributes missing changing an ALB's subnets or security groups (i.e. bumping az_count) goes through Set*, not Modify*
iam:UpdateAssumeRolePolicy, iam:UpdateRole, iam:UpdateRoleDescription missing any trust-policy / description drift on a cudly-* role
secretsmanager:UpdateSecretVersionStage, ListSecretVersionIds missing any secret version carrying a stage other than a lone AWSCURRENT
secretsmanager:ListSecrets inside an ARN-scoped statement dead grant no resource type, so unsatisfiable; dropped (nothing on the deploy path calls it)

Every Resource = "*" now documents which of the three reasons applies (no resource type / identifier-less request / opaque id), so a reviewer can tell a justified wildcard from a lazy one.

Security: IAMDenyModifyDeployRoleAndPolicies

iam:UpdateAssumeRolePolicy on role/cudly-* also matches cudly-terraform-deploy itself, so granting it would let the deploy role rewrite its own trust policy and make itself assumable by an arbitrary principal. The new Deny closes that, plus two pre-existing doors into the same escalation that the audit surfaced:

  • iam:AttachRolePolicy / iam:PutRolePolicy against the deploy role (attach AdministratorAccess to self);
  • iam:CreatePolicyVersion against policy/cudly-deploy-* (rewrite the deploy role's own permissions in place, without ever touching the role).

Denying only the role side would have looked complete without being complete. Verified no workload name can collide: all workload roles/policies derive from stack_name = "cudly-<env>-<hex>", while the Deny targets role/cudly-terraform-deploy and policy/cudly-deploy-* exactly. It costs the deploy path nothing, since those resources are declared in this bootstrap root and applied only by a human.

Deliberately NOT fixed here → LeanerCloud/cloud-commitments-platform#157

Scaled to what unblocks deploys, rather than shipping widenings I could not verify:

  • RDS proxy ARN scopes are dead as written. proxy:cudly-* is wrong on two counts (segment is db-proxy:, id is an opaque prx-…), and target-group:cudly-* on the opaque-id count. That makes four proxy actions unauthorizable and rds:ModifyDBProxyTargetGroup missing. Gated off by enable_rds_proxy = false, which is a literal at terraform/environments/aws/database.tf:28 and not a variable, so no tfvars can override it and all three proxy resources are count = 0 in every environment. The missing rds:ModifyDBProxyTargetGroup is therefore inert, and the three proxy Describes moved account-wide here are belt-and-braces rather than load-bearing. Not fixed because scoping an opaque id correctly needs a tag condition whose satisfiability must be verified, not assumed — which is exactly the mistake fix(iac/aws): grant kms:TagResource for KMS CreateKey tag-on-create #1671 made. The dead scopes are marked in place with a KNOWN DEAD SCOPES comment and a "do not add new proxy actions here" warning.
  • CloudFront cannot be enabled: cloudfront:TagResource is unsatisfiable at CreateDistributionWithTags (same shape as fix(iac/aws): grant kms:TagResource for KMS CreateKey tag-on-create #1671), and aws_cloudfront_origin_access_control has no permissions at all.
  • module "monitoring" is not instantiated and has no permissions. Several would be dead on arrival if added naively: SNS topic ARNs have no resource-type segment; CloudWatch dashboard ARNs have an empty region segment and a / separator; GuardDuty detector ids are opaque hex; the Security Hub hub ARN is literally hub/default; logs:*QueryDefinition*, xray:GetSamplingRules and guardduty:CreateDetector/ListDetectors have no resource type at all.
  • Wrong comment in the tree: terraform/modules/compute/aws/lambda/migration-alarm.tf:25-29 claims the bootstrap grants logs:PutMetricFilter. It does not.
  • rds:DescribeDBSnapshots is granted by no statement in any of the four policies. github-prod.tfvars:50 sets database_skip_final_snapshot = false, so a prod DB destroy takes a final snapshot and the provider waits on it. Destroy-path only, so it does not block this deploy. Filed as #1707; the scoping decision there (ARN-scoped vs account-wide) has to be made from the provider's actual request shape, not by analogy.
  • Opt-in leftovers: lambda:*FunctionConcurrency, acm:RemoveTagsFromCertificate, route53:CreateHostedZone/DeleteHostedZone/ChangeTagsForResource, logs:DeleteRetentionPolicy, and iam:PassRole for monitoring.rds.amazonaws.com if RDS enhanced monitoring is ever enabled.

Policy sizes (6144-char managed-policy limit)

Measured by rendering each jsonencode through Terraform, so these are the exact documents AWS sees:

Policy Before After Remaining
cudly-deploy-compute 5923 5933 211
cudly-deploy-compute-b 905 1290 4854
cudly-deploy-data 3548 4588 1556
cudly-deploy-networking 2675 2831 3313

policy_compute.tf was already at 5923 of 6144, so the only thing added to it is the 10-character StringEquals → StringEqualsIgnoreCase operator change. The new ECR and ELB statements go in policy_compute_b.tf, which exists for exactly this reason. No split was needed.

Verification

  • terraform fmt -check clean, terraform validate Success on all four policies.
  • Full pre-commit on the changed files: Terraform format / validate / lint (tflint) / Trivy config scanner / AWS secret scan all Passed. No --no-verify.
  • CI green on the first commit (AWS Sanity, Azure Sanity, CI Build & Test, pre-commit all success).
  • Every factual claim in the new comments checked against the tree: enable_nat_gateway = true (networking.tf:16), enable_flow_logs false/true/true across dev/staging/prod, enable_scheduled_tasks = true in all three tfvars, aws_iam_instance_profile.fck_nat referenced by the launch template, aws_flow_log.iam_role_arn, and the lowercase project_name = "cudly" that drives the CloudFront tag-case fix.
  • tfplan, .terraform/ and backend.hcl confirmed gitignored and absent from the diff.

Not verified against live AWS: no AWS mutation was attempted from here by design. The authoritative check is the bootstrap terraform plan below.

⚠️ Requires a manual bootstrap apply by a privileged human

terraform/environments/aws/ci-cd-permissions/ is a bootstrap-only root: it defines the deploy role's own permissions and is applied manually, never by a deploy workflow. Merging this does not unblock deploys on its own. After merge, with privileged credentials:

cd terraform/environments/aws/ci-cd-permissions
terraform init -backend-config=backend.hcl
terraform plan -out=tfplan
terraform apply tfplan

Expect changes to all four aws_iam_policy resources (a new default policy version each). Then re-run Deploy to AWS Lambda on main.

Summary by CodeRabbit

  • New Features

    • Expanded deployment capabilities for EC2, VPC Flow Logs, EventBridge, Auto Scaling, load balancing, ECS, RDS, and secrets management.
    • Added support for managing launch template versions, route-table associations, EC2 tags, load balancer settings, and container image scanning.
    • Improved compatibility with varied project tag capitalization and supporting EC2 resource authorization.
  • Security

    • Added safeguards preventing modification of deployment roles and policies.
    • Restricted EC2 instance launches to the approved instance type.

…ited grant gaps

terraform plan on main fails with AccessDenied on rds:DescribeDBInstances
against arn:aws:rds:us-east-1:*:db:*, even though the action was already
granted on arn:aws:rds:*:*:db:cudly-* and the instance name matches that
prefix.

The Terraform id of aws_db_instance is the DbiResourceId (db-<opaque>), not
the DB identifier. provider v5.100.0's findDBInstanceByID branches on the id
shape and, for a DbiResourceId, sends DescribeDBInstances with
Filters=[dbi-resource-id] and no DBInstanceIdentifier. RDS authorizes an
identifier-less DescribeDBInstances against the wildcard ARN db:*, which no
name-scoped grant can match. It is unconditional on every read: the
provider's retry with the plain identifier only fires on NotFound, and
AccessDenied is not NotFound, so the plan hard-fails.

Rather than add the one action that 403'd, this audits every resource and
data block under terraform/environments/aws/ and its modules against the
union of the four deploy policies, and fixes the gaps that block the deploy
path. It also names a third failure mode the previous four patches did not
account for: a grant can be present and still unusable because the request
carries no identifier, or because the real ARN uses a different segment or a
server-assigned opaque id.

Fixed here:

- Move rds:DescribeDBInstances and the DB proxy reads to a read-only
  Resource = "*" statement; every mutating RDS action stays ARN-scoped.
  Return rds:DescribeDBSubnetGroups to the scoped statement, since that call
  does carry a name and needs no widening.
- Add rds:ModifyDBSubnetGroup, which is required whenever az_count or the
  private subnet set changes.
- Extend the iam:PassedToService allowlist with ec2.amazonaws.com (fck-nat
  instance profile, on the deploy path in every environment),
  vpc-flow-logs.amazonaws.com (staging and prod) and events.amazonaws.com
  (scheduled ECS tasks). Each was a granted-but-unsatisfiable condition of
  the same shape as the kms:TagResource bug.
- Add iam:UpdateAssumeRolePolicy, iam:UpdateRole and iam:UpdateRoleDescription
  for trust-policy and description drift on cudly-* roles.
- Add secretsmanager:UpdateSecretVersionStage and
  secretsmanager:ListSecretVersionIds for secret versions carrying a stage
  other than a lone AWSCURRENT.
- Drop secretsmanager:ListSecrets, which has no resource type and so could
  never be satisfied inside an ARN-scoped statement. Nothing on the deploy
  path calls it.
- Add IAMDenyModifyDeployRoleAndPolicies. iam:UpdateAssumeRolePolicy on
  role/cudly-* also matches cudly-terraform-deploy, so granting it would let
  the deploy role rewrite its own trust policy. The Deny closes that along
  with two pre-existing doors into the same escalation: AttachRolePolicy /
  PutRolePolicy on the deploy role, and CreatePolicyVersion on the
  cudly-deploy-* policies. Denying only the role side would have looked
  complete without being complete.
- Document, for every Resource = "*", which of the three reasons applies (no
  resource type, identifier-less request, or opaque id) so a reviewer can
  tell a justified wildcard from a lazy one, and mark the known-dead RDS
  proxy ARN scopes in place so nothing new is added to them.

Deliberately not fixed here, enumerated in the issue and tracked separately:
the RDS proxy ARN scopes (proxy:/target-group: are unmatchable but the path
is gated off by enable_rds_proxy = false, and scoping an opaque id needs a
tag condition whose satisfiability has to be verified rather than assumed),
and the permissions for the monitoring module, cleanup-lambda and CloudFront,
none of which are instantiated on any deploy path today.

All four managed policies stay under the 6144-character limit; cudly-deploy-data
goes from 3548 to 4219 characters.

This root is bootstrap-only and is applied manually by a privileged human, so
merging does not by itself unblock deploys.

Closes #1698
@cristim cristim added triaged Item has been triaged priority/p0 Drop everything; same-day fix severity/critical Major harm when it happens urgency/now Drop other things impact/all-users Affects every user effort/m Days type/bug Defect labels Aug 3, 2026
@coderabbitai

coderabbitai Bot commented Aug 3, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

The CI/CD IAM policies add deployment permissions, protect deploy-role resources, correct RDS and Secrets Manager scopes, and expand compute and networking actions. CloudFront project-tag matching is now case-insensitive.

Changes

CI/CD IAM policy audit

Layer / File(s) Summary
IAM role and PassRole controls
terraform/environments/aws/ci-cd-permissions/policy_data.tf
Adds role update actions, expands PassRole services, enables additional service-linked roles, and denies deploy-role modification actions.
RDS and Secrets Manager resource scopes
terraform/environments/aws/ci-cd-permissions/policy_data.tf
Moves RDS discovery actions to account-wide read permissions, retains resource-scoped mutations, removes secretsmanager:ListSecrets, and adds secret-version actions.
Compute permission updates
terraform/environments/aws/ci-cd-permissions/policy_compute.tf, terraform/environments/aws/ci-cd-permissions/policy_compute_b.tf
Adds ECR image-scanning and load-balancer attribute actions. CloudFront project-tag matching is case-insensitive.
Networking permission updates
terraform/environments/aws/ci-cd-permissions/policy_networking.tf
Adds launch-template, tag, and route-table actions. The instance-type condition uses StringEqualsIfExists and retains the t4g.nano restriction.

Estimated code review effort: 4 (Complex) | ~45 minutes

Possibly related issues

  • #1698: The changes implement the linked audit fixes for RDS discovery, PassRole services, IAM role updates, and secret-version actions.
  • #1703: The changes address overlapping deploy-role permission gaps, including RDS proxy scopes and CloudFront tagging.
  • #1705: The explicit deploy-role and deploy-policy deny matches the issue’s IAM hardening objective.

Possibly related PRs

  • LeanerCloud/CUDly#524: Both changes modify RDS and ec2:RunInstances IAM scoping.
  • LeanerCloud/CUDly#568: Both changes expand policy_data.tf with deploy-role and RDS proxy permission changes.
  • LeanerCloud/CUDly#817: Both changes update Secrets Manager permissions by removing ListSecrets and adding secret-version actions.
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Linked Issues check ⚠️ Warning The PR fixes the listed authorization gaps, but it widens rds:DescribeDBSubnetGroups despite issue #1698 requiring that action to remain ARN-scoped. Keep rds:DescribeDBSubnetGroups in the scoped RDS statement unless an identifier-less call requiring Resource = "*" is documented and proven.
✅ Passed checks (4 passed)
Check name Status Explanation
Out of Scope Changes check ✅ Passed The compute, networking, data, edge, IAM, and Secrets Manager changes are all covered by issue #1698 and the stated audit objectives.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly identifies the RDS authorization fix and the broader audited permission updates described in the pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/deploy-role-rds-and-audit

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@terraform/environments/aws/ci-cd-permissions/policy_data.tf`:
- Around line 79-92: Update the iam:PassRole policy’s Resource entries to use
only the explicit role ARNs required by the listed service principals, replacing
the broad cudly-* role pattern. Map each ARN to the roles referenced by the NAT
launch template/Auto Scaling group, aws_flow_log DeliverLogsPermissionArn, and
EventBridge scheduled ECS task role_arn, while retaining the existing service
condition.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 4aad9adc-8f54-45c7-aaa8-cc2c8b51e88f

📥 Commits

Reviewing files that changed from the base of the PR and between 218f385 and 8a7a420.

📒 Files selected for processing (1)
  • terraform/environments/aws/ci-cd-permissions/policy_data.tf

Comment thread terraform/environments/aws/ci-cd-permissions/policy_data.tf
Second slice of the #1698 audit, covering the compute, container, edge and
networking resources. Same three defect classes as the RDS fix: an action
that is missing, an action gated on a condition that cannot hold, and an
action whose grant can never match the request.

Blocking on the deploy path:

- ec2:CreateLaunchTemplateVersion, ec2:ModifyLaunchTemplate and
  ec2:DeleteLaunchTemplateVersions. The fck-nat launch template sources its
  AMI from a most_recent = true lookup, so every upstream AMI republish takes
  the launch template's update path rather than create. Only
  Create/DeleteLaunchTemplate were granted, so the first AMI rotation after
  the ASG exists would have failed the apply.
- EC2RunInstancesFckNAT used StringEquals on ec2:InstanceType. AWS authorizes
  one RunInstances call against every resource type it touches, and
  ec2:InstanceType is only in the request context for the instance leg, so
  the condition evaluated false on the volume, network-interface,
  security-group, subnet, image and launch-template legs and denied the call
  as a whole. Switched to StringEqualsIfExists, which keeps the t4g.nano
  restriction exactly where the key exists. This is the operator the IAM
  condition-operators reference documents for this call.
- iam:CreateServiceLinkedRole for autoscaling.amazonaws.com,
  elasticloadbalancing.amazonaws.com and ecs.amazonaws.com. Service-linked
  roles are created lazily on first use of a service in an account/region, so
  these are invisible where the role already exists and a hard failure in a
  fresh region. Note ecs.application-autoscaling was already granted and is a
  different principal that does not cover plain ecs.

Latent, fixed while the file is open:

- ec2:DeleteTags, the other half of ec2:CreateTags. The provider removes
  dropped keys before adding new ones, so dropping any key from common_tags
  or default_tags on an existing EC2 resource failed the apply.
- ec2:ReplaceRouteTableAssociation, which the association's update path calls
  instead of Disassociate plus Associate.
- ecr:PutImageScanningConfiguration, whose sibling PutImageTagMutability was
  already granted.
- The elasticloadbalancing Set* family plus ModifyListenerAttributes.
  Changing a load balancer's subnets or security groups, which is what
  bumping az_count does, goes through Set* rather than Modify*.

- CloudFrontMutateTaggedOnly was gated on a case-sensitive StringEquals
  against "CUDly" while the distribution's Project tag resolves to the
  lowercase "cudly" via local.common_tags, so all four of its actions were
  silently denied. This is the same defect #1496 fixed on the KMS statements,
  left unfixed here because enable_cdn is false in every tfvars so nothing
  exercised it. It would also have killed the destroy path, since deleting a
  distribution requires UpdateDistribution to disable it first.

The ECR and ELB statements go in policy_compute_b.tf rather than
policy_compute.tf, which is the file's stated purpose: policy_compute is at
5933 of 6144 characters and would not hold them. Sizes after this change:
compute 5933, compute-b 1290, data 4588, networking 2831.

Refs #1698
…d statement

rds:DescribeDBSubnetGroups was the only rds:Describe* left in the
ARN-scoped RDSResourceScoped statement, and it is in exactly the same
position as the rds:DescribeDBInstances read that has been failing the
deploy: aws_db_subnet_group.main (terraform/modules/database/aws/main.tf:73)
is created in every environment, and a resource-scoped grant on a Describe
authorizes only the form that targets one named resource, while the
enumerate form carries no identifier and is authorized against the wildcard
ARN.

Worth recording why this is not the simpler claim that ARN-scoped Describes
are always dead, because that claim is false and would have justified the
wrong fix elsewhere. AWS's Service Authorization Reference confirms both
actions DO support resource types (db and subgrp), and the real names DO
match the patterns (cudly-<env>-<hex>-postgres against db:cudly-*,
cudly-<env>-<hex>-db-subnet against subgrp:cudly-*). The scoped grants were
syntactically valid; only the request shape differs. The production 403 on
DescribeDBInstances is the empirical proof that the scoped form did not
suffice.

Also records two facts established during review, so they are not
rediscovered:

- enable_rds_proxy is a literal false at
  terraform/environments/aws/database.tf:28, not a variable, so no tfvars
  can override it. All three proxy resources are count = 0 in every
  environment, which makes the proxy Describes here belt-and-braces rather
  than load-bearing.
- The RDSResourceScoped comment now says no rds:Describe* lives there at
  all, rather than naming only DescribeDBInstances.

Policy size is unchanged at 4588 of 6144 characters: the action moved
between statements rather than being added.

Refs #1698
@cristim

cristim commented Aug 3, 2026

Copy link
Copy Markdown
Member Author

Update: 8ac4a6dc5 moves rds:DescribeDBSubnetGroups account-wide

One-line change from adversarial review, plus two facts recorded so they are not rediscovered. No other behaviour changed.

What changed

rds:DescribeDBSubnetGroups was the only rds:Describe* still in the ARN-scoped RDSResourceScoped statement. Everything else had already moved to RDSDescribeAccountWide. It is now moved too.

It matters because it sits in exactly the same position as the read that has been failing this deploy all along: aws_db_subnet_group.main (terraform/modules/database/aws/main.tf:73) is created in every environment, and a resource-scoped grant on a Describe authorizes only the form targeting one named resource, while the enumerate form carries no identifier and is authorized against the wildcard ARN. Leaving it scoped would have primed the fifth turn of this treadmill: apply the bootstrap, watch the deploy fail on the next action, repeat.

Why this is not the simpler "ARN-scoped Describes are always dead"

Worth stating explicitly, because that claim is false and would justify the wrong fix elsewhere. The review checked AWS's machine-readable Service Authorization Reference and disproved it:

  • rds:DescribeDBInstances and rds:DescribeDBSubnetGroups genuinely do support resource types (db, subgrp).
  • The ARN patterns genuinely do match: cudly-<env>-<hex>-postgres → db:cudly-*, cudly-<env>-<hex>-db-subnet → subgrp:cudly-*.

So this was never a name-prefix bug, and the grants were syntactically valid. The enumerate-vs-target request shape is the remaining explanation, and the live 403 is the proof. That distinction is now in the statement's comment, because the wrong generalisation is the more expensive mistake.

Two facts recorded

  • RDS Proxy is not on the deploy path. enable_rds_proxy = false is a literal at terraform/environments/aws/database.tf:28, not a variable, so no tfvars can override it. All three proxy resources are count = 0 in every environment. The missing rds:ModifyDBProxyTargetGroup is therefore inert, and the three proxy Describes moved account-wide are belt-and-braces rather than load-bearing.
  • rds:DescribeDBSnapshots is granted by no statement in any of the four policies. github-prod.tfvars:50 sets database_skip_final_snapshot = false, so a prod DB destroy takes a final snapshot and the provider waits on it. Destroy-path only; does not block this deploy. Filed as #1707 rather than fixed here — and deliberately left undecided there whether it belongs ARN-scoped or account-wide, since that has to come from the provider's actual request shape rather than by analogy to this change.

Not touched

The CreateRole + AttachRolePolicy escalation is pre-existing, tracked in #1705, and its fix (an iam:PermissionsBoundary condition plus iam:PolicyARN on AttachRolePolicy) is a design change that does not belong in an outage fix. This PR remains net privilege-reducing versus its base.

Replied on the open CodeRabbit PassRole thread rather than leaving it silent: the finding is correct but pre-existing, and its suggested remedy is not implementable as written (the roles use name_prefix with an apply-time random_id, in a separate Terraform root that cannot interpolate from the workload state). Details there.

Verification

  • Policy size unchanged at 4588 / 6144 for cudly-deploy-data — the action moved between statements rather than being added, so this cost 0 characters, not the ~28 estimated.
  • terraform fmt -check clean, terraform validate Success.
  • Full pre-commit: Terraform format / validate / lint (tflint) / Trivy config scanner / AWS secret scan all Passed. No --no-verify.
  • Confirmed no rds:Describe* remains in RDSResourceScoped, and DescribeDBSubnetGroups appears exactly once in the action list.

Still requires the manual bootstrap apply by a privileged human described in the PR body; merging alone does not unblock deploys.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@terraform/environments/aws/ci-cd-permissions/policy_compute_b.tf`:
- Around line 114-131: Update the ELBFargateSetAttributes statement to scope
permissions by action: use listener ARNs restricted to cudly-* resources for
ModifyListenerAttributes, and load-balancer ARNs restricted to cudly-* resources
for SetIpAddressType, SetSecurityGroups, and SetSubnets. Replace Resource = "*"
with the appropriate action-specific resource statements while preserving the
existing Effect and actions.

In `@terraform/environments/aws/ci-cd-permissions/policy_networking.tf`:
- Around line 33-35: Scope the EC2 write permissions in the networking IAM
policy: constrain launch-template actions including CreateLaunchTemplateVersion,
DeleteLaunchTemplateVersions, and ModifyLaunchTemplate to the fck-nat
launch-template ARN pattern with ec2:ResourceTag/Project set to CUDly; restrict
DeleteTags to Terraform-managed EC2 resource types and approved aws:TagKeys; and
apply the same resource and tag-key boundary to the existing CreateTags
statement so it cannot tag arbitrary resources.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 6858c9eb-65f4-4624-9ca2-5da35f7e1c80

📥 Commits

Reviewing files that changed from the base of the PR and between 8a7a420 and 8ac4a6d.

📒 Files selected for processing (4)
  • terraform/environments/aws/ci-cd-permissions/policy_compute.tf
  • terraform/environments/aws/ci-cd-permissions/policy_compute_b.tf
  • terraform/environments/aws/ci-cd-permissions/policy_data.tf
  • terraform/environments/aws/ci-cd-permissions/policy_networking.tf

Comment on lines +114 to +131
{
# The ELB update path that policy_compute.tf's ELBFargate misses.
# ELBFargate grants Create/Delete plus the Modify* family, but changing
# a load balancer's subnets or security groups (which is what bumping
# az_count does) goes through the Set* family instead, and
# ModifyListenerAttributes is the writer whose reader
# (DescribeListenerAttributes) is already granted. Resource = "*"
# matches ELBFargate, whose ARNs are not name-scopeable.
Sid = "ELBFargateSetAttributes"
Effect = "Allow"
Action = [
"elasticloadbalancing:ModifyListenerAttributes",
"elasticloadbalancing:SetIpAddressType",
"elasticloadbalancing:SetSecurityGroups",
"elasticloadbalancing:SetSubnets",
]
Resource = "*"
},

@coderabbitai coderabbitai Bot Aug 3, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

mapfile -t files < <(
  fd --type f 'policy_compute(_b)?\.tf$' \
    terraform/environments/aws/ci-cd-permissions
)

test "${`#files`[@]}" -gt 0

rg -n -C 12 \
  'ELBFargate|ModifyListenerAttributes|SetIpAddressType|SetSecurityGroups|SetSubnets|Resource\s*=' \
  "${files[@]}"

Repository: LeanerCloud/CUDly

Length of output: 207


🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

files=()
while IFS= read -r f; do files+=("$f"); done < <(
  git ls-files 'terraform/environments/aws/ci-cd-permissions/policy_compute.tf' \
    'terraform/environments/aws/ci-cd-permissions/policy_compute_b.tf'
)
printf '%s\n' "${files[@]}"
test "${`#files`[@]}" -gt 0

for f in "${files[@]}"; do
  echo "===== $f ====="
  wc -l "$f"
  rg -n -C 18 \
    'ELBFargate|ModifyListenerAttributes|SetIpAddressType|SetSecurityGroups|SetSubnets|Resource\s*=|load_balancer|listener' \
    "$f" || true
done

Repository: LeanerCloud/CUDly

Length of output: 207


🌐 Web query:

AWS ELBv2 service authorization reference ModifyListenerAttributes SetIpAddressType SetSecurityGroups SetSubnets resource-level permissions resource types

💡 Result:

In AWS Elastic Load Balancing V2 (ELBv2), the actions ModifyListenerAttributes, SetIpAddressType, SetSecurityGroups, and SetSubnets all support resource-level permissions [1]. This means you can restrict these actions to specific load balancer resources in your IAM policies by specifying the appropriate Amazon Resource Name (ARN) in the Resource element [1][2]. According to the official AWS Service Authorization Reference for ELBv2, these actions require the following resource types [1]: 1. ModifyListenerAttributes: Requires the listener resource type (arn:aws:elasticloadbalancing:region:account-id:listener/load-balancer-id/target-group-id/listener-id) or load balancer resource type depending on the specific configuration. 2. SetIpAddressType, SetSecurityGroups, and SetSubnets: These actions require the load balancer resource type (e.g., arn:aws:elasticloadbalancing:region:account-id:loadbalancer/app/load-balancer-name/load-balancer-id or arn:aws:elasticloadbalancing:region:account-id:loadbalancer/gwy/load-balancer-name/load-balancer-id) [1]. In AWS IAM policies, if an action supports resource-level permissions, you should avoid using a wildcard ("*") and instead specify the ARN of the specific load balancer or listener you wish to permit access to [3][2]. If you specify a resource that is not valid for the action, the request will fail [3]. These actions are classified with an "Write" access level [1][4].

Citations:


🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

for f in \
  terraform/environments/aws/ci-cd-permissions/policy_compute.tf \
  terraform/environments/aws/ci-cd-permissions/policy_compute_b.tf
do
  if [ -f "$f" ]; then
    echo "===== $f ====="
    wc -l "$f"
    rg -n -C 18 \
      'ELBFargate|ModifyListenerAttributes|SetIpAddressType|SetSecurityGroups|SetSubnets|Resource\s*=|load_balancer|listener' \
      "$f" || true
  else
    echo "MISSING $f"
  fi
done

Repository: LeanerCloud/CUDly

Length of output: 24354


🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

for f in \
  terraform/environments/aws/ci-cd-permissions/policy_compute.tf \
  terraform/environments/aws/ci-cd-permissions/policy_compute_b.tf
do
  [ -f "$f" ] || continue
  echo "===== resources and ARN expressions in $f ====="
  rg -n -C 8 \
    'arn:|loadbalancer|listener|targetgroup|cudly|ELB|elasticloadbalancing' \
    "$f" || true
done

Repository: LeanerCloud/CUDly

Length of output: 14790


🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

echo "===== ELB resources and names ====="
rg -n -C 8 \
  'aws_lb|aws_lb_listener|aws_lb_target_group|load_balancer_type|name_prefix|name\s*=.*cudly|cudly-' \
  terraform modules -g '*.tf' || true

echo "===== ELB ARN references ====="
rg -n -C 5 \
  'elasticloadbalancing|load_balancer_arn|listener_arn|target_group_arn|arn:aws:elasticloadbalancing' \
  . -g '*.tf' || true

Repository: LeanerCloud/CUDly

Length of output: 50374


🌐 Web query:

site:docs.aws.amazon.com/service-authorization/latest/reference/list_elasticloadbalancingv2.html ModifyListenerAttributes SetIpAddressType SetSecurityGroups SetSubnets resource types

💡 Result:

For AWS ELBv2 IAM actions, the required resource types are:

Action Resource type
ModifyListenerAttributes listener/app*, listener/gwy*, listener/net*
SetIpAddressType loadbalancer/app/, loadbalancer/gwy/, loadbalancer/net/
SetSecurityGroups loadbalancer/app/, loadbalancer/net/
SetSubnets loadbalancer/app/, loadbalancer/gwy/, loadbalancer/net/

SetSecurityGroups additionally supports the elasticloadbalancing:SecurityGroup condition key; SetSubnets supports elasticloadbalancing:Subnet. [1]


🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

rg -l --glob '*.tf' \
  'resource[[:space:]]+"aws_lb"|resource[[:space:]]+"aws_lb_listener"|resource[[:space:]]+"aws_lb_target_group"' \
  terraform/modules terraform/environments |
while IFS= read -r f; do
  echo "===== $f ====="
  rg -n -C 12 \
    'resource[[:space:]]+"aws_lb"|resource[[:space:]]+"aws_lb_listener"|resource[[:space:]]+"aws_lb_target_group"|name_prefix|name[[:space:]]*=' \
    "$f"
done

Repository: LeanerCloud/CUDly

Length of output: 30667


Scope the ELB write permissions to deployment resources.

Use listener ARNs for ModifyListenerAttributes and load-balancer ARNs for the three Set* actions. Restrict them to the cudly-* deployment resources instead of Resource = "*".

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@terraform/environments/aws/ci-cd-permissions/policy_compute_b.tf` around
lines 114 - 131, Update the ELBFargateSetAttributes statement to scope
permissions by action: use listener ARNs restricted to cudly-* resources for
ModifyListenerAttributes, and load-balancer ARNs restricted to cudly-* resources
for SetIpAddressType, SetSecurityGroups, and SetSubnets. Replace Resource = "*"
with the appropriate action-specific resource statements while preserving the
existing Effect and actions.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pre-existing hardening advice, not a defect this PR introduces. Deferring with reasons, tracked in #1703.

First, a correction in CodeRabbit's favour, because the opposite claim was made during review of this PR and it is wrong: ELB ARNs are not opaque and these actions can be prefix-scoped. The ARN is arn:aws:elasticloadbalancing:<region>:<acct>:loadbalancer/app/<name>/<id> and the name is ours: aws_lb.main.name = local.name_prefix = "${var.stack_name}-fargate" (terraform/modules/compute/aws/fargate/main.tf:5,414), i.e. cudly-<env>-<hex>-fargate. So loadbalancer/app/cudly-*/* and listener/app/cudly-*/* would genuinely match. CodeRabbit's factual premise is correct and I am not disputing it.

The reason to defer is different, and it is that scoping these four actions changes the blast radius by approximately nothing.

The statement these belong to is ELBFargate in policy_compute.tf:250. It grants 19 elasticloadbalancing:* actions on Resource = "*", including DeleteLoadBalancer, DeleteTargetGroup, DeleteListener, CreateLoadBalancer, ModifyLoadBalancerAttributes and ModifyTargetGroup. That statement predates this PR and is untouched by it.

ELBFargateSetAttributes exists only because policy_compute.tf is at 5933 of 6144 characters and could not hold four more actions; it is a size-driven split of ELBFargate, which is exactly what policy_compute_b.tf's file header describes. So the proposal is to scope 4 of 23 sibling actions while the other 19 — including every destructive one — keep Resource = "*" in the adjacent file. An actor who can already DeleteLoadBalancer on any load balancer in the account does not need SetSubnets on it. The scoped version would read as hardened without being hardened, which is a failure mode this PR is specifically about.

The coherent change is to scope all 23 ELB actions together, in one statement, verified against live IAM. That is a hardening change to pre-existing grants and does not belong in a p0 outage fix where deploys have been down all session and the Fargate path cannot be exercised from here.

Two further notes, so the follow-up gets done correctly rather than quickly:

  • ModifyListenerAttributes and the three Set* actions take different resource types (listener vs loadbalancer), so they cannot share one Resource list without one of the two legs silently matching nothing. CodeRabbit is right to call for splitting them; that is a detail worth not losing.
  • Listener ARNs embed both the load-balancer id and the listener id (listener/app/<name>/<lb-id>/<listener-id>), both AWS-assigned, so the pattern has to be listener/app/cudly-*/*/* and not listener/app/cudly-*/*. A one-segment error there produces a grant that matches nothing and 403s on the next az_count change — the precise failure this PR exists to fix (rds:DescribeDBInstances was granted on arn:aws:rds:*:*:db:cudly-*, the name matched, and production still returned AccessDenied).

Tracked in #1703 with the ARN formats above so the next person does not re-derive them. Not resolving this thread.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cristim, acknowledged.

The finding is valid, but scoping only these four actions would not materially reduce the deploy role’s effective ELB permissions while ELBFargate retains wildcard access for the related destructive actions.

Deferring the complete ELB hardening to #1703 is appropriate for this outage-focused PR. The follow-up should scope all related ELB actions together and use separate statements for listener and load-balancer resource types. The listener pattern must use listener/app/cudly-*/*/*.

🐇


✏️ Learnings added
Learnt from: cristim
Repo: LeanerCloud/CUDly PR: 1699
File: terraform/environments/aws/ci-cd-permissions/policy_compute_b.tf:114-131
Timestamp: 2026-08-03T14:34:24.808Z
Learning: In `terraform/environments/aws/ci-cd-permissions`, `ELBFargateSetAttributes` in `policy_compute_b.tf` is a managed-policy-size split from the pre-existing wildcard-scoped `ELBFargate` statement in `policy_compute.tf`. Do not treat scoping only the four split ELB actions as meaningful hardening while the related `ELBFargate` actions remain `Resource = "*"`. Track the coherent all-action ELB scoping work in GitHub issue `#1703`. `ModifyListenerAttributes` requires listener ARNs and must use the pattern `listener/app/cudly-*/*/*`; `SetIpAddressType`, `SetSecurityGroups`, and `SetSubnets` require load-balancer ARNs.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.

You are interacting with an AI system.

Comment on lines +33 to +35
"ec2:CreateLaunchTemplateVersion",
"ec2:DeleteLaunchTemplateVersions",
"ec2:ModifyLaunchTemplate",

@coderabbitai coderabbitai Bot Aug 3, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟠 Major | 🏗️ Heavy lift

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Find the tag contract used by networking resources.
rg -n -C 4 'default_tags|tags\s*=|Project\s*=|ManagedBy\s*=' terraform

# Find deploy-role attachments and related IAM resource constraints.
rg -n -C 4 'cudly-deploy-networking|aws_iam_policy|aws_iam_role_policy_attachment|assume_role' terraform

Repository: LeanerCloud/CUDly

Length of output: 50376


🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '--- policy ---'
cat -n terraform/environments/aws/ci-cd-permissions/policy_networking.tf

printf '%s\n' '--- networking launch-template and tags ---'
rg -n -C 8 'launch_template|delete_tags|DeleteTags|Project|ManagedBy|default_tags|tags\s*=' \
  terraform/modules/networking/aws terraform/environments/aws

printf '%s\n' '--- CI/CD permission files and role references ---'
rg -n -C 5 'VPCNetworking|CreateLaunchTemplateVersion|DeleteLaunchTemplateVersions|ModifyLaunchTemplate|ec2:DeleteTags|cudly-deploy-networking|policy_networking|aws_iam' \
  terraform/environments/aws/ci-cd-permissions

Repository: LeanerCloud/CUDly

Length of output: 50376


🌐 Web query:

AWS EC2 service authorization reference CreateLaunchTemplateVersion DeleteLaunchTemplateVersions ModifyLaunchTemplate DeleteTags resource types condition keys aws:ResourceTag

💡 Result:

For managing Amazon EC2 launch templates and resource tags, authorization is controlled through IAM policies by specifying the appropriate API actions, resource types, and condition keys [1][2][3]. API Actions and Resource Authorization The actions CreateLaunchTemplateVersion, DeleteLaunchTemplateVersions, and ModifyLaunchTemplate all support resource-level permissions [1][2]. This allows you to restrict access to specific launch templates by including their Amazon Resource Name (ARN) in the Resource element of your IAM policy [3]. - CreateLaunchTemplateVersion: Allows creating a new version of a specified launch template [4]. - DeleteLaunchTemplateVersions: Allows deleting one or more versions of a specified launch template [5]. - ModifyLaunchTemplate: Allows changing the default version associated with a launch template [6]. When granting access to these actions, note that they can indirectly impact security. For example, modifying a launch template (ModifyLaunchTemplate) or creating a new version (CreateLaunchTemplateVersion) can allow a user to associate an IAM role with an instance if they can point the template to a different version that contains an instance profile, even without explicit iam:PassRole permission [1][2]. DeleteTags Authorization The DeleteTags action is used to remove tags from existing resources [7]. It supports resource-level permissions, meaning you can control which resources a user can remove tags from [8][9]. Condition Keys and aws:ResourceTag The aws:ResourceTag condition key is used to control access to a resource based on the tags currently attached to it (Attribute-Based Access Control, or ABAC) [9]. - Usage: In an IAM policy, you can use the Condition element to allow or deny an action based on whether the target resource has specific tags [9]. For example, "StringEquals": { "aws:ResourceTag/environment": "production" } [9]. - Applicability: While aws:ResourceTag is widely supported for actions that act on tagged resources, its applicability depends on whether the specific API action supports resource-level permissions [3][9]. You should always verify if a specific action supports resource-level permissions in the official AWS Service Authorization Reference [3]. - Security Caution: If you use aws:ResourceTag to control access, you should generally deny users the ability to modify those tags (i.e., deny ec2:CreateTags and ec2:DeleteTags on those resources) to prevent them from bypassing your security controls [9][10]. For the most authoritative and current mappings of actions to their supported resource types and condition keys, always consult the official AWS Service Authorization Reference for Amazon EC2 [3][9].

Citations:


🏁 Script executed:

#!/bin/bash
set -euo pipefail

python3 - <<'PY'
from urllib.request import urlopen
from html.parser import HTMLParser
import re

url = "https://docs.aws.amazon.com/service-authorization/latest/reference/list_amazonec2.html"
html = urlopen(url, timeout=20).read().decode("utf-8", "replace")

for action in (
    "CreateLaunchTemplateVersion",
    "DeleteLaunchTemplateVersions",
    "ModifyLaunchTemplate",
    "DeleteTags",
):
    pos = html.find(f">{action}<")
    if pos < 0:
        pos = html.find(action)
    print(f"\n--- {action} ---")
    print(re.sub(r"<[^>]+>", " ", html[max(0, pos-1500):pos+2500]))
PY

Repository: LeanerCloud/CUDly

Length of output: 2441


🏁 Script executed:

#!/bin/bash
set -euo pipefail

curl -k -fsSL \
  'https://docs.aws.amazon.com/service-authorization/latest/reference/list_amazonec2.html' |
python3 - <<'PY'
import sys
import re
from html import unescape

html = sys.stdin.read()
for action in (
    "CreateLaunchTemplateVersion",
    "DeleteLaunchTemplateVersions",
    "ModifyLaunchTemplate",
    "DeleteTags",
):
    matches = list(re.finditer(rf">{re.escape(action)}<", html))
    print(f"\n--- {action} ({len(matches)} matches) ---")
    for match in matches[:1]:
        row = html[max(0, match.start()-2500):match.end()+3500]
        row = unescape(re.sub(r"<[^>]+>", " ", row))
        row = re.sub(r"\s+", " ", row)
        print(row)
PY

Repository: LeanerCloud/CUDly

Length of output: 356


🏁 Script executed:

#!/bin/bash
set -euo pipefail

curl -k -fsSL \
  'https://docs.aws.amazon.com/service-authorization/latest/reference/list_amazonec2.html' \
  -o /tmp/aws-ec2-service-authorization.html

python3 - <<'PY'
import re
from html import unescape

with open("/tmp/aws-ec2-service-authorization.html", encoding="utf-8") as f:
    html = f.read()

for action in (
    "CreateLaunchTemplateVersion",
    "DeleteLaunchTemplateVersions",
    "ModifyLaunchTemplate",
    "DeleteTags",
):
    matches = list(re.finditer(rf">{re.escape(action)}<", html))
    print(f"\n--- {action} ({len(matches)} matches) ---")
    for match in matches[:1]:
        row = html[max(0, match.start()-2500):match.end()+3500]
        row = unescape(re.sub(r"<[^>]+>", " ", row))
        row = re.sub(r"\s+", " ", row)
        print(row)
PY

Repository: LeanerCloud/CUDly

Length of output: 328


🏁 Script executed:

#!/bin/bash
set -euo pipefail

python3 - <<'PY'
import re

with open("/tmp/aws-ec2-service-authorization.html", encoding="utf-8") as f:
    html = f.read()

print("bytes:", len(html))
for needle in ("LaunchTemplate", "DeleteTags", "CreateLaunch", "ModifyLaunch"):
    print(f"\n--- {needle} ---")
    for match in list(re.finditer(needle, html, re.I))[:5]:
        snippet = html[max(0, match.start()-300):match.end()+500]
        print(re.sub(r"\s+", " ", snippet))
PY

Repository: LeanerCloud/CUDly

Length of output: 255


🌐 Web query:

site:docs.aws.amazon.com/service-authorization/latest/reference/list_amazonec2.html "CreateLaunchTemplateVersion" "ec2:ResourceTag"

💡 Result:

The ec2:ResourceTag condition key is a global condition key used in AWS Identity and Access Management (IAM) policies to restrict access based on tags attached to an AWS resource [1][2]. It is not a direct permission itself, but rather a tool to enforce fine-grained access control when calling API actions such as ec2:CreateLaunchTemplateVersion [1][2]. When you include ec2:ResourceTag in an IAM policy, you can control whether a user or role is permitted to execute CreateLaunchTemplateVersion on specific EC2 launch templates based on the tags currently assigned to those templates [1][2]. For example, you can allow a user to call CreateLaunchTemplateVersion only if the target launch template has a specific tag: { "Effect": "Allow", "Action": [ "ec2:CreateLaunchTemplateVersion" ], "Resource": "*", "Condition": { "StringLike": { "ec2:ResourceTag/Project": "DataPipeline" } } } In this configuration, the CreateLaunchTemplateVersion action will be permitted only if the resource being modified matches the specified tag criteria [1]. This pattern is commonly used in managed policies to ensure that automated services or users can only modify resources that they are explicitly authorized to manage [1][2].

Citations:


Scope EC2 write permissions to CUDly resources.

Restrict the launch-template actions to the fck-nat launch-template ARN pattern and require ec2:ResourceTag/Project = CUDly. Restrict ec2:DeleteTags to the EC2 resource types Terraform manages and allowed aws:TagKeys. Apply an equivalent boundary to the existing unscoped ec2:CreateTags; otherwise the role can add Project = CUDly to arbitrary resources and bypass the tag condition. (docs.aws.amazon.com)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@terraform/environments/aws/ci-cd-permissions/policy_networking.tf` around
lines 33 - 35, Scope the EC2 write permissions in the networking IAM policy:
constrain launch-template actions including CreateLaunchTemplateVersion,
DeleteLaunchTemplateVersions, and ModifyLaunchTemplate to the fck-nat
launch-template ARN pattern with ec2:ResourceTag/Project set to CUDly; restrict
DeleteTags to Terraform-managed EC2 resource types and approved aws:TagKeys; and
apply the same resource and tag-key boundary to the existing CreateTags
statement so it cannot tag arbitrary resources.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pre-existing hardening advice, not a defect this PR introduces. Deferring with reasons, tracked in #1703 — but the third point in this finding is the most valuable thing CodeRabbit has said on this PR and it is being recorded, not dismissed.

Taking the three parts separately, because they are not equally actionable.

The ec2:CreateTags point is correct and is a precondition, not an addendum

otherwise the role can add Project = CUDly to arbitrary resources and bypass the tag condition

This is right, and it is the reason the first suggestion cannot be applied on its own. ec2:CreateTags is granted on Resource = "*" with no condition (policy_networking.tf:29, pre-existing). Introduce an ec2:ResourceTag/Project gate anywhere in EC2 and that unconditioned grant immediately defeats it: the role self-tags the target, then passes its own gate. It is the same hole the KMSTagOnCreate statement in policy_compute_b.tf guards against with its Null condition on aws:ResourceTag/Project, added for exactly this reason.

So adopting suggestion 1 without suggestion 3 would produce a gate that looks like a boundary and is not one. That makes this a single coupled change, not a quick win.

And that coupled change is the problem: ec2:CreateTags is on the critical path of every EC2 resource create in this configuration — VPC, subnets, route tables, both gateways, all seven security groups, and the launch template all create with tags, authorized through ec2:CreateAction tag-on-create semantics. Getting an aws:TagKeys or resource-type boundary marginally wrong there does not degrade gracefully; it fails terraform apply at the first resource. Doing that inside a p0 outage fix, with deploys down all session and no ability to test against live IAM from here, is not a trade I will make.

The launch-template tag gate is fragile in a way this repo has been bitten by three times

Two concrete problems with ec2:ResourceTag/Project = CUDly on the launch-template actions:

  1. The ARN cannot be prefix-scoped. Launch template ARNs are arn:aws:ec2:<region>:<acct>:launch-template/lt-<opaque>; the id is AWS-assigned and contains nothing of ${var.stack_name}-fck-nat- (terraform/modules/networking/aws/main.tf:214). So "the fck-nat launch-template ARN pattern" has to be launch-template/* plus a tag condition, and the whole boundary rests on the tag.

  2. The tag value is load-bearing and unstable. aws_launch_template.fck_nat sets no top-level tags argument — only tag_specifications, which tag the launched instances and volumes, not the launch template itself. The LT therefore inherits Project = "CUDly" from the provider's default_tags (terraform/environments/aws/main.tf:35), and a case-sensitive StringEquals "CUDly" would match today. But every sibling resource that sets resource-level tags gets local.common_tags, whose Project = var.project_name is the lowercase "cudly" (main.tf:100, github-dev.tfvars:9), which overrides default_tags. The moment anyone adds tags = var.tags to the launch template — the most natural change in the world, since every neighbouring resource has it — the gate silently stops matching and the next AMI rotation 403s.

That is not hypothetical. It is the exact bug fixed in #1496 (kms:ScheduleKeyDeletion denied on a CUDly-owned key), re-found in #1514, and found a third time in this very PR on CloudFrontMutateTaggedOnly, which had been silently denying all four of its actions. Any new tag gate in this repo must use StringEqualsIgnoreCase, as KMSMutateTaggedOnly, KMSReadTaggedOnly, KMSTagOnCreate and now CloudFrontMutateTaggedOnly all do. The suggestion as written specifies the case-sensitive form.

ec2:DeleteTags is the one part I will own as new

The three launch-template actions and ReplaceRouteTableAssociation are additions to a pre-existing Resource = "*" statement of ~50 EC2 actions, and are read/update operations on resources the role already fully manages. ec2:DeleteTags is different in kind: it adds the ability to remove tags from any EC2 resource in the account, which the role could not previously do.

In context this is a small delta — the same statement already grants ec2:DeleteVpc, ec2:DeleteSubnet, ec2:DeleteSecurityGroup and ec2:DeleteRouteTable on Resource = "*", so tag removal is nowhere near the top of this role's blast radius — and it is required: the provider's tag-update path calls DeleteTags for removed keys before CreateTags for added ones, so dropping any key from common_tags fails the apply without it. But it should be scoped by resource type and aws:TagKeys in the same pass that scopes CreateTags, and I have recorded it that way rather than letting it ride as "consistent with its neighbours".

Where this goes

Folded into #1703 as a single coupled work item: scope ec2:CreateTags and ec2:DeleteTags together by resource type and aws:TagKeys, then and only then add the launch-template boundary, using StringEqualsIgnoreCase and launch-template/* rather than a name pattern. Not resolving this thread.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cristim, agreed. The finding combined pre-existing hardening work with the new ec2:DeleteTags permission too broadly.

A launch-template tag condition is not a security boundary while unrestricted ec2:CreateTags can self-apply the required tag. The CreateTags, DeleteTags, and launch-template changes must stay coupled.

Your deferral to #1703 is appropriate for this outage-fix PR. The follow-up should:

  • Scope ec2:CreateTags and ec2:DeleteTags together.
  • Include the required EC2 resource types and approved aws:TagKeys.
  • Use launch-template/*, because launch-template IDs are AWS-assigned.
  • Use StringEqualsIgnoreCase for Project, because the repository has both CUDly and cudly values.

The new ec2:DeleteTags access remains the relevant delta in this PR. #1703 records the required remediation and its prerequisites.

🐇 🔍


✏️ Learnings added
Learnt from: cristim
Repo: LeanerCloud/CUDly PR: 1699
File: terraform/environments/aws/ci-cd-permissions/policy_networking.tf:33-35
Timestamp: 2026-08-03T14:34:34.581Z
Learning: In `terraform/environments/aws/ci-cd-permissions/policy_networking.tf`, do not add an `ec2:ResourceTag/Project` authorization boundary for launch-template mutations until `ec2:CreateTags` is constrained. An unrestricted `ec2:CreateTags` grant allows the deploy role to self-tag arbitrary EC2 resources and bypass the tag boundary. Scope `ec2:CreateTags` and `ec2:DeleteTags` together by supported EC2 resource types and approved `aws:TagKeys`.

Learnt from: cristim
Repo: LeanerCloud/CUDly PR: 1699
File: terraform/environments/aws/ci-cd-permissions/policy_networking.tf:33-35
Timestamp: 2026-08-03T14:34:34.581Z
Learning: In the AWS Terraform CI/CD IAM policies, launch-template ARNs use AWS-assigned opaque `lt-*` identifiers and cannot be constrained by an application name prefix. If a launch-template policy boundary is added, use `arn:aws:ec2:<region>:<account>:launch-template/*` with a tag condition after tag mutation permissions are safely scoped.

Learnt from: cristim
Repo: LeanerCloud/CUDly PR: 1699
File: terraform/environments/aws/ci-cd-permissions/policy_networking.tf:33-35
Timestamp: 2026-08-03T14:34:34.581Z
Learning: In the AWS Terraform configuration, `Project` tag values can be both `CUDly` from provider `default_tags` and `cudly` from `local.common_tags`. Any IAM tag condition for this value must use `StringEqualsIgnoreCase` to avoid Terraform authorization failures when resource-level tags override provider default tags.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.

You are interacting with an AI system.

@cristim

cristim commented Aug 3, 2026

Copy link
Copy Markdown
Member Author

Summary for whoever applies this

This PR requires a privileged human to apply the bootstrap manually — merging it changes nothing on its own. This comment is meant to be readable without reconstructing the change from the diff.

What was broken

terraform plan on main failed with AccessDenied on rds:DescribeDBInstances against arn:aws:rds:us-east-1:...:db:* — an action the deploy role already had, scoped to db:cudly-*, with an instance name (cudly-dev-426fc8af-postgres) that matches that prefix.

The Terraform id of aws_db_instance is the DbiResourceId (db-<opaque>), not the DB identifier. Provider v5.100.0's findDBInstanceByID branches on the id shape and, for a DbiResourceId, sends DescribeDBInstances with Filters=[dbi-resource-id] and DBInstanceIdentifier nil. RDS authorizes an identifier-less Describe against the wildcard ARN, which no name-scoped grant can match. It fires on every read: the provider's retry with the plain identifier only triggers on NotFound, and AccessDenied is not NotFound.

Why this PR is bigger than that one action

The same grant had already been patched four times, each time adding whatever 403'd next (#1496, #1514, #1671, #1698). So this audits every resource/data block under terraform/environments/aws/ and its modules against all four policies. It found that a grant can be present and still fail in three ways, only the first of which is visible by reading the policy:

  1. Missing action.
  2. Unsatisfiable condition — cannot hold at call time (tag-on-create; a case-sensitive "CUDly" against a tag that is lowercase cudly; a condition key that exists on only one of the resource legs a single call authorizes against).
  3. Unmatchable resource — the request carries no identifier, or the real ARN uses a different segment or a server-assigned opaque id.

All three were found again, live, in code nobody had looked at. The fixes are listed in the PR body.

rds:DescribeDBSubnetGroups — the likely next 403

8ac4a6dc5 moved this one action account-wide. It was the only rds:Describe* left ARN-scoped, and aws_db_subnet_group.main (terraform/modules/database/aws/main.tf:73) is created in every environment — the same call shape as the read that was already failing. Left alone it would very likely have been the fifth turn of this treadmill: apply the bootstrap, watch the deploy fail on the next action, repeat.

Worth being precise about why, because the tempting generalisation is wrong and would justify bad fixes elsewhere. AWS's Service Authorization Reference confirms both actions do support resource types (db, subgrp), and the real names do match the patterns. The scoped grants were syntactically valid. Only the request shape differs: a resource-scoped grant authorizes the form targeting one named resource, while the enumerate form is authorized against the wildcard. That reasoning is now in the policy comment itself, so it survives the next tidy-up.

Two facts the review established

  • RDS Proxy is off the deploy path. enable_rds_proxy = false is a literal at terraform/environments/aws/database.tf:28, not a variable, so no tfvars can override it and all three proxy resources are count = 0 everywhere. The missing rds:ModifyDBProxyTargetGroup is therefore inert, and the proxy Describes moved account-wide are belt-and-braces rather than load-bearing.
  • rds:DescribeDBSnapshots is granted by no statement in any of the four policies. github-prod.tfvars:50 sets database_skip_final_snapshot = false, so a prod DB destroy takes a final snapshot and the provider waits on it. Destroy-path only; does not block this deploy. Filed as #1707, deliberately leaving the ARN-scoped-vs-account-wide decision to be made from the provider's actual request shape rather than by analogy to this PR.

⚠️ This PR is net privilege-REDUCING

Important for anyone reviewing what they are about to apply. It adds actions, but it also adds IAMDenyModifyDeployRoleAndPolicies, an explicit Deny that beats any Allow and closes:

  • iam:UpdateAssumeRolePolicy on the deploy role (which this PR grants on cudly-* roles, and which would otherwise let the role rewrite its own trust policy so an arbitrary principal could assume it);
  • pre-existing: iam:AttachRolePolicy / iam:PutRolePolicy against the deploy role — attach AdministratorAccess to self;
  • pre-existing: iam:CreatePolicyVersion against policy/cudly-deploy-* — rewrite the deploy role's own permissions in place, without ever touching the role.

Denying only the role side would have looked complete without being complete. Adversarial review confirmed PassRole provably cannot reach the deploy role or anything that can assume it.

What it does not close: the deploy role can still CreateRole a fresh cudly-* role, attach a broad policy to it, and pass it. That is pre-existing, tracked as #1705, and its correct fix is an iam:PermissionsBoundary condition on CreateRole plus iam:PolicyARN on AttachRolePolicy — bounding what any created role can do, rather than enumerating which roles exist. That is a design change and does not belong in an outage fix.

CodeRabbit threads

All three have written replies; none resolved, none silenced. Note CodeRabbit's status on this PR reads "Review rate limited", so its pass is not a complete review either way.

  • PassRole scoping — correct but pre-existing, and the remedy is not implementable as written: the roles use name_prefix with an apply-time random_id, in a separate Terraform root that cannot interpolate from workload state.
  • ELB Set* on Resource = "*" — correct, and I confirmed ELB ARNs are not opaque (loadbalancer/app/cudly-*/* would match). Deferred because ELBFargate already grants 19 ELB actions on "*" including every destructive one; scoping 4 of 23 siblings reads as hardened without being hardened.
  • EC2 tag / launch-template scoping — the most valuable of the three. Its ec2:CreateTags point is correct and is a precondition: that grant is unconditioned on "*", so any new ec2:ResourceTag gate is defeated by self-tagging. Deferred because bounding CreateTags touches the critical path of every EC2 create in the config, and because the suggested StringEquals "CUDly" is the exact case bug fixed three times already.

Both new threads folded into #1703 with the verified ARN formats and the required ordering, so the follow-up is done correctly rather than quickly.

Gates

terraform fmt -check clean, terraform validate Success, full pre-commit (tflint, Trivy, secret scan) passed, no --no-verify, CI 4/4 green on 8ac4a6dc5. Policy sizes: compute 5933, compute-b 1290, data 4588, networking 2831 — all under 6144, measured by rendering each jsonencode through Terraform. No AWS mutation was attempted from here; the authoritative check is the bootstrap terraform plan in the PR body.

@cristim
cristim merged commit 802b4b1 into main Aug 3, 2026
20 checks passed
@cristim
cristim deleted the fix/deploy-role-rds-and-audit branch August 3, 2026 15:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

effort/m Days impact/all-users Affects every user priority/p0 Drop everything; same-day fix severity/critical Major harm when it happens triaged Item has been triaged type/bug Defect urgency/now Drop other things

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(iac/aws): deploy role blocked on rds:DescribeDBInstances; full audit of cudly-terraform-deploy grants

1 participant