feat(iam): grant karpenter ec2 instance-status and capacity-reservation reads - #194
Merged
Merged
Conversation
…on reads Karpenter 1.14.1 (cloud-infrastructure-applications v0.59.0) registers an instance-status health-check controller unconditionally. Without ec2:DescribeInstanceStatus it logs a single AccessDenied line and returns an empty reconciler.Result, so it never requeues and the checks stay off until the pod restarts. Also grants ec2:DescribeCapacityReservations. It is not called today -- the provider short-circuits when an EC2NodeClass has no capacityReservationSelectorTerms -- but without it, adopting capacity reservations fails EC2NodeClass reconcile, leaving ConditionTypeCapacityReservationsReady unset and stopping provisioning on that NodePool. Granting the read now avoids needing a module release mid capacity crunch. Both actions are read-only and stay under the existing aws:RequestedRegion condition. ec2:DescribePlacementGroups and the capacity-reservation/ placement-group RunInstances ARNs from the upstream v1.14.1 policy are deliberately omitted: the ARNs are unconditioned launch capability we do not use, and cluster placement groups are single-AZ, conflicting with our multi-AZ topology. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
maxsxu
approved these changes
Sep 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Karpenter 1.14.1 is becoming the default via cloud-infrastructure-applications v0.59.0 (chart bump PR). Two EC2 read actions in the upstream v1.14.1 controller policy are missing from this module.
1.
ec2:DescribeInstanceStatus— currently broken on every AWS pool member.1.12.0 added an instance-status health-check controller. It is registered unconditionally — there is no feature gate and no dependency on
settings.interruptionQueue, which we do not set:https://github.com/aws/karpenter-provider-aws/blob/v1.14.1/pkg/controllers/controllers.go#L115
On
AccessDeniedit logs once and returns an emptyreconciler.Result, so it never requeues:https://github.com/aws/karpenter-provider-aws/blob/v1.14.1/pkg/controllers/interruption/instancestatus_controller.go#L98
The observable symptom is a single line at startup, after which the controller is inert until the pod restarts:
2.
ec2:DescribeCapacityReservations— latent, but fails closed on provisioning.Not called in our current configuration:
provider.Listbuilds zero queries when anEC2NodeClasshas nocapacityReservationSelectorTermsand returns before touching the API.https://github.com/aws/karpenter-provider-aws/blob/v1.14.1/pkg/providers/capacityreservation/provider.go#L77
The failure mode on adoption is not a log line.
nodeclass.CapacityReservation.ReconcilecallsListon every reconcile; an error there meansConditionTypeCapacityReservationsReadyis never set, theEC2NodeClassgoesNotReady, and that NodePool stops provisioning nodes:https://github.com/aws/karpenter-provider-aws/blob/v1.14.1/pkg/controllers/nodeclass/capacityreservation.go#L54
Since
featureGates.reservedCapacityis BETA and enabled by default in the chart, and capacity reservations are the natural lever in quota-constrained regions, granting the read up front avoids discovering a slow fix path — module release, CEBC bump, provision apply — at the moment the capability is needed.Modifications
Two read-only actions added to the
AllowRegionalReadActionsstatement ofdata.aws_iam_policy_document.karpenter. Both remain scoped by the statement's existingaws:RequestedRegioncondition. No mutating actions, no resource-ARN changes, no new statements.Deliberately not included from the upstream v1.14.1 policy:
ec2:DescribePlacementGroupsPlacementGroupSelector() == nilreturns before the API call. Cluster placement groups are single-AZ, which conflicts with our multi-AZ broker/bookie topology, so this is capability we have no path to using.capacity-reservation/*andplacement-group/*ARNs onAllowScopedEC2InstanceAccessActionsarc-zonal-shift:GetManagedResourcesettings.enableZonalShift, which isfalsein our values; the operator builds a no-op provider otherwise. Also not permitted byStreamNativeCloudPermissionBoundary, so enabling it is a separate, wider change.Verifying this change
terraform fmt -checkis clean. Action list verified free of duplicates and the statement's region condition is unchanged.The policy delta was derived by diffing the upstream controller policy (published only as the CloudFormation template in the getting-started docs) between the version we run today and v1.14.1, then confirming each action's call path in the 1.14.1 Go source rather than trusting the template alone. That source reading is what produced the three omissions above.
This IAM change has not yet been applied to a live cluster — it is inert until a provision stage applies it. The chart side was verified independently on a test CE (us-east-2, k8s v1.34.9-eks) in the chart bump PR, which is where the
DescribeInstanceStatusdenial was observed and where capacity-reservation and placement-group reads were confirmed not to be called — Karpenter provisioned nodes normally with neither permission granted. Post-merge verification is that the startup denial no longer appears and the controller reaches a steady 60s poll.Deployment note
Granting
ec2:DescribeInstanceStatusturns on a behavior that is currently off: the controller polls every 60s for impaired instance/system status and scheduled maintenance events, then cordons and drains affected nodes.InstanceStatusDryRundefaults tofalse, so remediation is live, not metrics-only. It routes through Karpenter's normal NodeClaim termination path and so respects PDBs — it is not a preemption — but it is a new source of node replacement on Pulsar node pools and worth watching on first rollout.Documentation
no-need-docInternal IAM policy for a cluster add-on; no user-facing surface.