Skip to content

feat(iam): grant karpenter ec2 instance-status and capacity-reservation reads - #194

Merged
maxsxu merged 1 commit into
masterfrom
dpappa/karpenter-1-14-iam-read-actions
Sep 5, 2026
Merged

feat(iam): grant karpenter ec2 instance-status and capacity-reservation reads#194
maxsxu merged 1 commit into
masterfrom
dpappa/karpenter-1-14-iam-read-actions

Conversation

@dpappa

@dpappa dpappa commented Sep 4, 2026

Copy link
Copy Markdown
Member

Motivation

Karpenter 1.14.1 is becoming the default via cloud-infrastructure-applications v0.59.0 (chart bump PR). Two EC2 read actions in the upstream v1.14.1 controller policy are missing from this module.

1. ec2:DescribeInstanceStatus — currently broken on every AWS pool member.

1.12.0 added an instance-status health-check controller. It is registered unconditionally — there is no feature gate and no dependency on settings.interruptionQueue, which we do not set:

https://github.com/aws/karpenter-provider-aws/blob/v1.14.1/pkg/controllers/controllers.go#L115

On AccessDenied it logs once and returns an empty reconciler.Result, so it never requeues:

https://github.com/aws/karpenter-provider-aws/blob/v1.14.1/pkg/controllers/interruption/instancestatus_controller.go#L98

The observable symptom is a single line at startup, after which the controller is inert until the pod restarts:

ec2:DescribeInstanceStatus permission is not allowed, update the IAM policy and
restart the Karpenter deployment to enable instance status health checks

2. ec2:DescribeCapacityReservations — latent, but fails closed on provisioning.

Not called in our current configuration: provider.List builds zero queries when an EC2NodeClass has no capacityReservationSelectorTerms and returns before touching the API.

https://github.com/aws/karpenter-provider-aws/blob/v1.14.1/pkg/providers/capacityreservation/provider.go#L77

The failure mode on adoption is not a log line. nodeclass.CapacityReservation.Reconcile calls List on every reconcile; an error there means ConditionTypeCapacityReservationsReady is never set, the EC2NodeClass goes NotReady, and that NodePool stops provisioning nodes:

https://github.com/aws/karpenter-provider-aws/blob/v1.14.1/pkg/controllers/nodeclass/capacityreservation.go#L54

Since featureGates.reservedCapacity is BETA and enabled by default in the chart, and capacity reservations are the natural lever in quota-constrained regions, granting the read up front avoids discovering a slow fix path — module release, CEBC bump, provision apply — at the moment the capability is needed.

Modifications

Two read-only actions added to the AllowRegionalReadActions statement of data.aws_iam_policy_document.karpenter. Both remain scoped by the statement's existing aws:RequestedRegion condition. No mutating actions, no resource-ARN changes, no new statements.

Deliberately not included from the upstream v1.14.1 policy:

Upstream item Why omitted
ec2:DescribePlacementGroups Same short-circuit shape — PlacementGroupSelector() == nil returns before the API call. Cluster placement groups are single-AZ, which conflicts with our multi-AZ broker/bookie topology, so this is capability we have no path to using.
capacity-reservation/* and placement-group/* ARNs on AllowScopedEC2InstanceAccessActions Real launch capability, and unconditioned in the upstream template — it would let the controller launch into any capacity reservation in the account, including reservations bought for unrelated workloads. Grant at adoption, alongside the selector terms that need it.
arc-zonal-shift:GetManagedResource Gated on settings.enableZonalShift, which is false in our values; the operator builds a no-op provider otherwise. Also not permitted by StreamNativeCloudPermissionBoundary, so enabling it is a separate, wider change.

Verifying this change

  • Make sure that the change passes the CI checks.

terraform fmt -check is clean. Action list verified free of duplicates and the statement's region condition is unchanged.

The policy delta was derived by diffing the upstream controller policy (published only as the CloudFormation template in the getting-started docs) between the version we run today and v1.14.1, then confirming each action's call path in the 1.14.1 Go source rather than trusting the template alone. That source reading is what produced the three omissions above.

This IAM change has not yet been applied to a live cluster — it is inert until a provision stage applies it. The chart side was verified independently on a test CE (us-east-2, k8s v1.34.9-eks) in the chart bump PR, which is where the DescribeInstanceStatus denial was observed and where capacity-reservation and placement-group reads were confirmed not to be called — Karpenter provisioned nodes normally with neither permission granted. Post-merge verification is that the startup denial no longer appears and the controller reaches a steady 60s poll.

Deployment note

Granting ec2:DescribeInstanceStatus turns on a behavior that is currently off: the controller polls every 60s for impaired instance/system status and scheduled maintenance events, then cordons and drains affected nodes. InstanceStatusDryRun defaults to false, so remediation is live, not metrics-only. It routes through Karpenter's normal NodeClaim termination path and so respects PDBs — it is not a preemption — but it is a new source of node replacement on Pulsar node pools and worth watching on first rollout.

Documentation

  • no-need-doc

    Internal IAM policy for a cluster add-on; no user-facing surface.

…on reads

Karpenter 1.14.1 (cloud-infrastructure-applications v0.59.0) registers an
instance-status health-check controller unconditionally. Without
ec2:DescribeInstanceStatus it logs a single AccessDenied line and returns an
empty reconciler.Result, so it never requeues and the checks stay off until
the pod restarts.

Also grants ec2:DescribeCapacityReservations. It is not called today -- the
provider short-circuits when an EC2NodeClass has no
capacityReservationSelectorTerms -- but without it, adopting capacity
reservations fails EC2NodeClass reconcile, leaving
ConditionTypeCapacityReservationsReady unset and stopping provisioning on
that NodePool. Granting the read now avoids needing a module release mid
capacity crunch.

Both actions are read-only and stay under the existing aws:RequestedRegion
condition. ec2:DescribePlacementGroups and the capacity-reservation/
placement-group RunInstances ARNs from the upstream v1.14.1 policy are
deliberately omitted: the ARNs are unconditioned launch capability we do not
use, and cluster placement groups are single-AZ, conflicting with our
multi-AZ topology.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@dpappa
dpappa requested a review from a team as a code owner September 4, 2026 17:54
@github-actions github-actions Bot added the no-need-doc This pr does not need any document label Sep 4, 2026
@maxsxu
maxsxu merged commit 2265e5f into master Sep 5, 2026
5 checks passed
@maxsxu
maxsxu deleted the dpappa/karpenter-1-14-iam-read-actions branch September 5, 2026 06:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

no-need-doc This pr does not need any document

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants