From 0c73fc442acf91869a781062fc0a56d5aba9453c Mon Sep 17 00:00:00 2001 From: Teddy Kahwaji Date: Thu, 3 Sep 2026 11:33:37 -0400 Subject: [PATCH 1/6] [DOCS-14545] Add AWS Lambda to the AWS integration instrumentation guides Expand the AWS integration instrumentation guide and technical reference to cover AWS Lambda functions alongside Amazon EC2 instances: - Per-workload prerequisites, including supported Lambda runtimes, architectures, package type, partition, and the layer limit - Per-workload IAM permission tables, with change-notification permissions split out as shared - What Datadog changes on a function, and the workload identity federation model that avoids writing an API key into the function - Lambda reconciliation, layer version rollout, and checkpoint-based uninstall - Lambda troubleshooting, including exclusivity with remote instrumentation Generalize the aws-agent-installation shortcode to describe permissions per workload. Co-Authored-By: Claude Opus 5 (1M context) --- ...-agent-installation-technical-reference.md | 139 ++++++++++---- .../guide/aws-agent-installation.md | 178 ++++++++++++++---- .../shortcodes/aws-agent-installation.en.md | 15 +- 3 files changed, 259 insertions(+), 73 deletions(-) diff --git a/hugo/content/en/integrations/guide/aws-agent-installation-technical-reference.md b/hugo/content/en/integrations/guide/aws-agent-installation-technical-reference.md index 52ae5cd6df4..c21c177ff76 100644 --- a/hugo/content/en/integrations/guide/aws-agent-installation-technical-reference.md +++ b/hugo/content/en/integrations/guide/aws-agent-installation-technical-reference.md @@ -1,11 +1,11 @@ --- -title: How Agent Installation through the AWS Integration Works -description: "Understand how Datadog installs and maintains the Datadog Agent on Amazon EC2 through the AWS integration: the AWS resources created, the installation mechanism, the security model, and the Agent lifecycle." +title: How Datadog Instrumentation through the AWS Integration Works +description: "Understand how Datadog instruments Amazon EC2 instances and AWS Lambda functions through the AWS integration: the AWS resources created, the instrumentation mechanism, the security model, and the reconciliation behavior." private: true # TODO(DOCS-14545): remove at v1 rollout to publish, at the same time as the setup guide this page links to; also add the nextlink entry under "AWS guides" in hugo/content/en/integrations/guide/_index.md at that time further_reading: - link: "https://docs.datadoghq.com/integrations/guide/aws-agent-installation/" tag: "Documentation" - text: "Install the Datadog Agent through the AWS Integration" + text: "Install Datadog instrumentation through the AWS Integration" - link: "https://docs.datadoghq.com/integrations/amazon_web_services/" tag: "Documentation" text: "AWS Integration" @@ -14,23 +14,28 @@ further_reading: text: "Fleet Automation" --- -This page explains how Datadog installs and maintains the Agent on Amazon EC2 through the AWS integration. For setup instructions and the permissions Datadog requires, see [Install the Datadog Agent through the AWS Integration][1]. +This page explains how Datadog instruments and maintains your AWS workloads through the AWS integration. For setup instructions and the permissions Datadog requires, see [Install Datadog instrumentation through the AWS Integration][1]. -
This page covers the Amazon EC2 experience only.
+This page covers Amazon EC2 instances and AWS Lambda functions. Amazon EKS is not supported. ## AWS resources that Datadog creates +### Created once, by the CloudFormation stack + The CloudFormation template you launch creates the following resources one time, in a single stack: | Resource | Name | Purpose | |---|---|---| | EventBridge connection | `datadog-agent-resource-update-intake-connection` | Holds your Datadog API and application keys so events can be sent to Datadog | | EventBridge API destination | `datadog-agent-resource-update-intake-destination` | Sends events to `https://api./api/unstable/instrumenter/events` (capped at 10 events per second) | -| EventBridge rule | `datadog-agent-resource-update-rule-ec2` | Notifies Datadog when a covered instance changes | +| EventBridge rule | `datadog-agent-resource-update-rule-ec2` | Notifies Datadog when a covered instance changes. Created when you select the EC2 workload | +| EventBridge rule | `datadog-agent-resource-update-rule-lambda` | Notifies Datadog when a covered function changes. Created when you select the Lambda workload | | IAM role | auto-named | Lets EventBridge send events to the `datadog-agent-resource-update-intake-destination` API destination | | IAM role | `datadog-eventbridge-cross-region-role` | Lets other regions forward events to your primary region | -Datadog creates the following resources as needed, at install time: +The stack also attaches the IAM permissions for the workloads you selected to your AWS integration role. A Lambda-only selection receives no EC2 permissions, and no IAM permissions boundary. + +### Created as needed, for EC2 instances | Resource | Name | Purpose | |---|---|---| @@ -42,9 +47,15 @@ Datadog creates the following resources as needed, at install time: Datadog does not create S3 buckets, event buses, log groups, or SSM parameters, and does not tag your instances. -## How Agent installation works +### Created as needed, for Lambda functions + +Datadog creates no AWS resources for Lambda instrumentation. The only change is to the configuration of the functions your rule covers. Datadog does not create secrets, IAM roles, or SSM documents for Lambda, and does not tag your functions. -After you save an installation rule, Datadog resolves the query you defined into a fixed list of covered instances. Datadog then runs the following sequence against each one. For prerequisites, including supported platforms, see [Prerequisites][2] in the setup guide. +## How instrumentation works + +After you save an instrumentation rule, Datadog resolves the query you defined into the set of covered resources, then runs the following sequence against each one. For prerequisites, including supported platforms and runtimes, see [Prerequisites][2] in the setup guide. + +### On Amazon EC2 1. Datadog checks that each covered instance is running, on a supported platform, and reachable by AWS Systems Manager. 2. When an instance has no IAM instance profile, Datadog creates one so Systems Manager can reach it. When an instance already has one, Datadog adds the SSM policy and the scoped secret-read policy to the existing role. @@ -54,73 +65,129 @@ After you save an installation rule, Datadog resolves the query you defined into Datadog does not reboot or restart your instances. The only service Datadog touches is the Datadog Agent itself, which is started on install and stopped on uninstall. Your applications and other services are untouched. -### Instances that Datadog excludes +### On AWS Lambda + +Lambda instrumentation runs entirely from Datadog. Datadog does not deploy a function into your account and does not run `datadog-ci` there. + +1. Datadog reads the function's current configuration and tags, and checks that it meets the [Lambda prerequisites][3]. +2. Datadog checks whether the function is already instrumented. A function carrying Datadog layers, a Datadog handler, or Datadog environment variables that Datadog did not apply is skipped, as is a function managed by [remote instrumentation][4]. The two cases are reported separately. +3. Datadog resolves the Datadog layer versions for the function's runtime, architecture, region, and AWS partition. Layer versions come from a pinned set that advances with Datadog's layer releases, so an installation is reproducible rather than tracking whatever is newest at that moment. +4. Datadog computes the complete desired configuration and records a checkpoint of exactly what it changes, before changing anything. +5. Datadog reconciles the intake authentication mapping for the function's execution role. See [How Lambda telemetry is authenticated](#how-lambda-telemetry-is-authenticated). +6. Datadog calls `lambda:UpdateFunctionConfiguration` once, submitting the complete layer list and environment map. The checkpoint is marked applied only after AWS reports success. + +A Lambda update is a replace-style operation: the submitted layer list and environment map become the new configuration. Datadog therefore computes the full desired state rather than appending to it, preserving your existing layers and environment variables. The update carries the function's revision ID, so a change made in your account between Datadog's read and write causes the update to fail rather than overwrite it. + +### What Datadog changes on a function + +| Change | Applies to | +|---|---| +| Adds the Datadog extension layer (`Datadog-Extension` or `Datadog-Extension-ARM`) | All supported runtimes | +| Adds the matching Datadog tracing layer | Node.js, Python, Ruby, Java, and .NET | +| Sets `DD_SITE` and `DD_ORG_UUID` | All supported runtimes | +| Redirects the handler to the Datadog handler and moves the original into `DD_LAMBDA_HANDLER` | Node.js and Python | +| Sets `AWS_LAMBDA_EXEC_WRAPPER` to `/opt/datadog_wrapper` | Java and .NET | + +Datadog does not change function code, memory size, timeout, VPC configuration, concurrency, or any other function setting. + +### Resources that Datadog excludes -Datadog automatically excludes: +On EC2, Datadog excludes: - Instances that are not running - EKS worker nodes - ECS container instances - Instances that already have a non-Datadog-installed Agent +On Lambda, Datadog excludes: + +- Container image functions, and functions on an unsupported runtime or architecture +- Functions outside the commercial `aws` partition +- Lambda@Edge replicas, and the functions they replicate +- Functions already instrumented by you or by remote instrumentation +- Functions that already set `AWS_LAMBDA_EXEC_WRAPPER` to a non-Datadog wrapper +- Functions where adding the Datadog layers would exceed the AWS five-layer limit + ## Security, auditing, and change control ### How Datadog gets access -Datadog uses the same cross-account IAM role as the AWS integration, authenticated with an external ID. Datadog receives short-lived, temporary credentials, and each type of work (reading EC2, managing IAM, sending commands) uses a separately scoped credential session rather than one broad session. Datadog does not store any long-lived AWS keys. +Datadog uses the same cross-account IAM role as the AWS integration, authenticated with an external ID. Datadog receives short-lived, temporary credentials, and each type of work (reading EC2, managing IAM, sending commands, updating functions) uses a separately scoped credential session rather than one broad session. Datadog does not store any long-lived AWS keys. ### Auditing Datadog's actions -Every action Datadog takes is a standard AWS API call, so all actions appear in AWS CloudTrail. Everything Datadog creates is identifiable by name: resources are prefixed with `datadog-`, secrets are stored under `/datadog/ec2-instrumenter/`, and IAM roles use the immutable path `/datadog-ec2-instrumenter/`. Because an IAM path cannot be edited after creation, the path cannot be silently changed. On-instance command results appear in the Systems Manager Run Command history. +Every action Datadog takes is a standard AWS API call, so all actions appear in AWS CloudTrail. Everything Datadog creates is identifiable by name: resources are prefixed with `datadog-`, secrets are stored under `/datadog/ec2-instrumenter/`, and IAM roles use the immutable path `/datadog-ec2-instrumenter/`. Because an IAM path cannot be edited after creation, the path cannot be silently changed. On-instance command results appear in the Systems Manager Run Command history. Lambda configuration changes appear as `UpdateFunctionConfiguration` events attributed to your AWS integration role. -### How the API key is handled +### How the API key is handled on EC2 The API key is stored in your own Secrets Manager, encrypted at rest. Only the secret's Amazon Resource Name (ARN) is passed in the SSM command; the key itself never appears in command parameters or in CloudTrail. The instance reads the secret with its own IAM role, restricted to a single path. Datadog stores only a reference to the key internally, not the key itself. -### Who can change installations +### How Lambda telemetry is authenticated + +Lambda instrumentation stores no Datadog credential in your account. The Datadog extension authenticates with the function's AWS execution identity through workload identity federation, using the `DD_ORG_UUID` and `DD_SITE` values Datadog sets on the function. No Datadog API key, secret ARN, or KMS-encrypted key is written into the function's configuration. + +For that authentication to succeed, Datadog maintains a mapping between the execution role's STS session pattern and your organization's managed rotation key. Datadog reconciles this mapping before submitting a function update. Mapping creation is idempotent, and Datadog accepts only an exact existing mapping rather than an overlapping one. + +Because a single execution role is often shared across functions, Datadog creates these mappings but does not remove them on uninstall. Removing a mapping for a shared role could break another function that still depends on it. + +### Who can change instrumentation - **In AWS**: Access is governed by your own IAM policies. Removing the cross-account permissions stops Datadog immediately. -- **In Datadog**: Viewing installation rules requires the **Hosts Read** permission. Creating, editing, or deleting rules requires the **Agent Install** permission. Rule changes are rate-limited. +- **In Datadog**: Viewing instrumentation rules requires the **Hosts Read** permission. Creating, editing, or deleting rules requires the **Agent Install** permission. Rule changes are rate-limited. ### Guardrails -- Datadog never removes an Agent it did not install. -- Datadog tracks which instances it installed an Agent on, so it cleans up only its own work. -- When some regions cannot be listed, Datadog skips cleanup for that pass rather than risk uninstalling in bulk. +- Datadog never removes instrumentation it did not install. +- Datadog tracks which resources it instrumented, so it cleans up only its own work. +- On EC2, when some regions cannot be listed, Datadog skips cleanup for that pass rather than risk uninstalling in bulk. +- On Lambda, Datadog restores a function from the checkpoint it recorded, so an uninstall reverses exactly the change Datadog made. +- A failure is scoped to the individual resource. One throttled or invalid resource does not cause Datadog to reprocess resources that already succeeded. -## Agent lifecycle and reconciliation +## How Datadog maintains instrumentation -### Rule coverage is fixed at save time +### Continuous reconciliation -A rule covers the list of instances it resolved to when you saved it, and Datadog does not instrument anything outside that list. Instances launched later are not picked up automatically. To cover them, update the rule, which re-resolves your query against your current fleet. +Datadog continuously maintains the state you define on the covered resources: -### How Datadog keeps covered instances in sync +- A full reconciliation runs hourly per AWS account. Reconciliation restores instrumentation if it goes missing, retries anything that failed, and cleans up resources that no longer exist. +- Change events forwarded from your account let Datadog react to covered resources within minutes, instead of waiting for the hourly pass. For EC2, these come from the CloudFormation stack's EventBridge rule. For Lambda, the `datadog-agent-resource-update-rule-lambda` rule forwards function create, configuration update, tag, and untag events. +- On EC2, already-installed instances are re-verified about once per day rather than every hour, to avoid unnecessary activity. +- On Lambda, the hourly scan checks each covered function against the layer versions Datadog deploys, and does per-function work only for functions that need a change. A fleet already on current layer versions produces no per-function activity, which keeps Datadog well clear of Lambda control-plane throttling. -Datadog continuously maintains the state you define on the covered instances: +### How Lambda functions pick up new layer versions -- A full reconciliation runs hourly per AWS account. Reconciliation reinstalls the Agent if it goes missing, retries anything that failed, and cleans up instances that no longer exist. -- Already-installed instances are re-verified about once per day rather than every hour, to avoid unnecessary activity. -- Change events from the CloudFormation stack let Datadog react to covered instances within minutes, instead of waiting for the hourly pass. +Datadog resolves layer versions from a pinned set on every reconciliation, rather than from whatever the function was first instrumented with. When Datadog releases new layer versions and that pinned set advances, the next reconciliation sees that a covered function's layers no longer match the desired versions and updates it. Your functions therefore move forward with Datadog's layer releases without any action from you. + +A Lambda configuration update that is still in progress is left alone and retried shortly afterward, so Datadog does not race a change already being applied. + + + +### Rule coverage on EC2 is fixed at save time + +An EC2 rule covers the list of instances it resolved to when you saved it, and Datadog does not instrument anything outside that list. Instances launched later are not picked up automatically. To cover them, update the rule, which re-resolves your query against your current fleet. ### What happens when you edit a rule -Datadog re-resolves your query and compares it against the previous list. Instances no longer covered have the Agent uninstalled. Newly covered instances have the Agent installed. Deleting a rule uninstalls the Agent from everything the rule covered. +Datadog re-resolves your query and compares it against the previous set. Resources no longer covered have instrumentation removed. Newly covered resources are instrumented. Deleting a rule removes instrumentation from everything the rule covered. -### Terminated or stopped instances +### Terminated, stopped, or deleted resources -Datadog detects terminated instances on the next hourly pass and cleans up the IAM resources it created for them. Datadog leaves stopped instances alone until they return. +On EC2, Datadog detects terminated instances on the next hourly pass and cleans up the IAM resources it created for them. Datadog leaves stopped instances alone until they return. On Lambda, a deleted function drops out of reconciliation on the next pass. -### When an install fails +### When instrumentation fails -Datadog retries with an increasing delay (1 hour, then 2 hours, up to once per day) and continues retrying. Missing-permission problems appear as an issue on the **AWS integration tile** and on the Fleet install page. +Datadog retries with an increasing delay (1 hour, then 2 hours, up to once per day) and continues retrying. Problems that need your action, such as a missing permission or a function at the layer limit, are reported and stop being retried indefinitely. Missing-permission problems appear as an issue on the **AWS integration tile** and on the Fleet install page.
-When someone manually removes the Agent from a covered instance, the next reconciliation reinstalls it. The rule is the source of truth. To stop coverage, change the rule. +When someone removes instrumentation from a covered resource by hand, the next reconciliation restores it. The rule is the source of truth. To stop coverage, change the rule.
-## Uninstall the Agent +## Uninstall + +To uninstall, remove resources from a rule, edit the rule's query, or delete the rule. -Uninstalling removes the Datadog Agent, the `/etc/datadog-agent` and `/opt/datadog-agent` directories on Linux (or performs an MSI uninstall on Windows), and any IAM role or instance profile Datadog created for that instance. To uninstall, remove instances from a rule, edit the rule's query, or delete the rule. +- **EC2**: Uninstalling removes the Datadog Agent, the `/etc/datadog-agent` and `/opt/datadog-agent` directories on Linux (or performs an MSI uninstall on Windows), and any IAM role or instance profile Datadog created for that instance. +- **Lambda**: Uninstalling removes the Datadog layers Datadog added and restores the environment variables and handler recorded in the checkpoint. Datadog validates the checkpoint against the function's current configuration first, so it does not remove a layer or variable it did not add. The intake authentication mapping for the execution role is left in place, because the role may be shared with other functions. ## Further reading @@ -128,3 +195,5 @@ Uninstalling removes the Datadog Agent, the `/etc/datadog-agent` and `/opt/datad [1]: https://docs.datadoghq.com/integrations/guide/aws-agent-installation/ [2]: https://docs.datadoghq.com/integrations/guide/aws-agent-installation/#prerequisites +[3]: https://docs.datadoghq.com/integrations/guide/aws-agent-installation/#aws-lambda-functions +[4]: https://docs.datadoghq.com/serverless/aws_lambda/remote_instrumentation/ diff --git a/hugo/content/en/integrations/guide/aws-agent-installation.md b/hugo/content/en/integrations/guide/aws-agent-installation.md index 1ad15de4e33..6465ae315ff 100644 --- a/hugo/content/en/integrations/guide/aws-agent-installation.md +++ b/hugo/content/en/integrations/guide/aws-agent-installation.md @@ -1,11 +1,11 @@ --- -title: Install the Datadog Agent through the AWS Integration -description: "Install and manage the Datadog Agent on your Amazon EC2 instances directly from the AWS integration, without connecting to each host or running per-host scripts." +title: Install Datadog Instrumentation through the AWS Integration +description: "Instrument your Amazon EC2 instances and AWS Lambda functions directly from the AWS integration, without connecting to each host or redeploying each function." private: true # TODO(DOCS-14545): remove at v1 rollout to publish; also add the nextlink entry under "AWS guides" in hugo/content/en/integrations/guide/_index.md at that time further_reading: - link: "https://docs.datadoghq.com/integrations/guide/aws-agent-installation-technical-reference/" tag: "Documentation" - text: "How Agent installation through the AWS integration works" + text: "How Datadog instrumentation through the AWS integration works" - link: "https://docs.datadoghq.com/integrations/amazon_web_services/" tag: "Documentation" text: "AWS Integration" @@ -21,29 +21,76 @@ further_reading: - link: "https://docs.datadoghq.com/agent/configuration/" tag: "Documentation" text: "Agent Configuration" +- link: "https://docs.datadoghq.com/serverless/aws_lambda/instrumentation/" + tag: "Documentation" + text: "Instrumenting AWS Lambda" --- ## Overview -The [AWS integration][1] collects metrics, events, and logs from Amazon CloudWatch without installing anything on your hosts. Installing the Datadog Agent adds telemetry from inside your AWS workloads that CloudWatch alone can't provide, including host-level metrics, distributed traces (APM), live processes, and detailed logs. +The [AWS integration][1] collects metrics, events, and logs from Amazon CloudWatch without installing anything on your resources. Adding Datadog instrumentation collects telemetry from inside your AWS workloads that CloudWatch alone can't provide, including host-level metrics, distributed traces (APM), live processes, and detailed logs. + +You can instrument your AWS workloads directly from Datadog, without connecting to each host or redeploying each function. Enable instrumentation while you set up the AWS integration, or at any time afterward. + +## Supported workloads -You can deploy the Datadog Agent to your Amazon EC2 instances directly from Datadog, without connecting to each host or running per-host scripts. Enable Agent installation while you set up the AWS integration, or at any time afterward. +| Workload | What Datadog installs | +|---|---| +| Amazon EC2 instances | The Datadog Agent | +| AWS Lambda functions | The Datadog Lambda extension, and the Datadog tracing layer matching the function's runtime | + +Amazon EKS is not supported. For Lambda functions, this guide covers instrumentation managed from Datadog. To instrument functions with a Lambda function deployed in your own account instead, see [Remote instrumentation for AWS Lambda][9]. -Amazon EKS is not supported. +
A Lambda function can be managed by only one Datadog instrumentation product. Datadog skips any function that remote instrumentation already manages, and skips any function you instrumented yourself.
## Prerequisites -Before you begin, confirm the following: +For all workloads, confirm the following: + +- **CloudFormation access**: You can approve a CloudFormation stack in the target AWS account. Instrumentation deploys a stack in your account, so you (or a teammate) need permission to review and create it. For the required permissions and why they're needed, see the [Required AWS permissions](#required-aws-permissions) section. +- **Datadog permissions**: Viewing instrumentation rules requires the **Hosts Read** permission. Creating, editing, or deleting rules requires the **Agent Install** permission. + +### Amazon EC2 instances -- **CloudFormation access**: You can approve a CloudFormation stack in the target AWS account. Installation deploys a stack in your account, so you (or a teammate) need permission to review and create it. For the required permissions and why they're needed, see the [Required AWS permissions](#required-aws-permissions) section. - **SSM Agent**: The [AWS Systems Manager (SSM) Agent][2] must already be present on the target instances. Datadog installs the Agent through SSM and can't install the SSM Agent for you, so instances built from custom AMIs without the SSM Agent are not eligible. Datadog flags these instances so you can address them. - **Supported platforms**: Linux (x86_64 and arm64) and Windows (x86_64). macOS and Windows on arm64 are not supported. +### AWS Lambda functions + +- **Resource collection**: [Resource collection][10] must be enabled on the AWS integration. Datadog uses it to list your functions and preview which ones a rule matches. +- **AWS partition**: The commercial `aws` partition only. Functions in AWS GovCloud or the AWS China partitions are not supported, because Lambda instrumentation authenticates through cloud-based authentication, which those partitions don't support. +- **Package type**: Zip functions only. Container image functions are not supported, because Datadog instrumentation is distributed as Lambda layers, which container image functions can't use. +- **Architecture**: `x86_64` or `arm64`. A function must report a single architecture. +- **Layer count**: AWS limits a function to five layers. Datadog adds up to two, so a function that already carries four or more layers is not eligible. +- **Supported runtimes**: + + | Runtime | Versions | + |---|---| + | Node.js | 16.x, 18.x, 20.x, 22.x, 24.x | + | Python | 3.8, 3.9, 3.10, 3.11, 3.12, 3.13, 3.14 | + | Ruby | 3.2, 3.3, 3.4, 4.0 | + | Java | 8, 8 (Amazon Linux 2), 11, 17, 21, 25 | + | .NET | 6, 8, 10 | + | OS-only (`provided.al2`, `provided.al2023`) | Datadog adds the extension layer only, with no tracing layer | + +Datadog marks any function that doesn't meet these conditions as ineligible in the rule preview, so you can see what is excluded before you apply a rule. + ## Required AWS permissions {{% aws-agent-installation %}} -Datadog uses each of these permissions for a specific task: +The following sections list the permissions for each workload. The CloudFormation stack grants only the permissions for the workloads you selected. + +### Change notification permissions + +These permissions apply to any workload that reacts to AWS resource changes: + +| Permission | Why Datadog needs it | +|---|---| +| `events:PutRule`, `events:PutTargets`, `events:DescribeRule`, `events:ListTargetsByRule`, `events:RemoveTargets`, `events:DeleteRule` | Set up the change notifications that let Datadog react to resource changes | +| `iam:GetRole`, `iam:PassRole` | Read and pass the EventBridge cross-region role. Both are restricted to the `datadog-eventbridge-cross-region-role` role, and `iam:PassRole` is further restricted to the EventBridge service | + +### Amazon EC2 permissions | Permission | Why Datadog needs it | |---|---| @@ -55,66 +102,105 @@ Datadog uses each of these permissions for a specific task: | `iam:CreateRole`, `iam:CreateInstanceProfile`, `iam:AddRoleToInstanceProfile`, `iam:AttachRolePolicy`, `iam:PutRolePolicy`, `iam:PassRole`, `ec2:AssociateIamInstanceProfile`, and the matching `Get` and `List` reads | Give an instance the minimum access it needs in case it does not have an IAM role: reachable by Systems Manager, and able to read its own API key secret | | `iam:Detach*`, `iam:Delete*`, `iam:RemoveRoleFromInstanceProfile`, `ec2:Disassociate*`, `ec2:DescribeIamInstanceProfileAssociations` | Cleanly undo the resources above when you uninstall | | `ecs:ListClusters`, `ecs:ListContainerInstances` | Recognize Amazon Elastic Container Service (ECS) container instances so Datadog skips them (they are handled at the cluster level) | -| `events:PutRule`, `events:PutTargets`, `events:RemoveTargets`, `events:DeleteRule` | Set up the change notifications that let Datadog react to instance changes | `iam:CreateRole` and `iam:PassRole` are the most sensitive grants. `iam:CreateRole` is restricted to role names matching `datadog-ec2-instrumenter/datadog-ssm-*` in your account, and `iam:PassRole` is further restricted to the Amazon EC2 service. +### AWS Lambda permissions + +| Permission | Why Datadog needs it | +|---|---| +| `lambda:ListFunctions` | Find the functions in your account and region | +| `cloudfront:ListDistributions` | Identify Lambda@Edge functions so Datadog skips them | +| `lambda:GetFunctionConfiguration`, `lambda:ListTags` | Read a function's configuration and tags to check which functions match your rule | +| `lambda:UpdateFunctionConfiguration` | Add the Datadog layers and environment variables, and remove them on uninstall | +| `lambda:GetLayerVersion` | AWS reauthorizes every layer submitted during a function update, including your own unchanged layers | + +Lambda instrumentation needs no Secrets Manager, Systems Manager, or IAM write permissions. Function reads and updates are restricted to Lambda functions in your own account. + ## How it works -Agent installation is based on an **installation rule**: an AWS account paired with a query that describes which EC2 instances to cover. Saving a rule resolves the query into a fixed list of instances. Datadog then installs the Agent on each one, inside your own account: +Instrumentation is based on an **instrumentation rule**: an AWS account paired with a query that describes which resources to cover. Datadog resolves the query into the set of covered resources, instruments each one inside your own account, and keeps them instrumented: -1. You select the EC2 instances to cover, or opt in to all eligible instances. -1. Datadog resolves your selection into a list of covered instances and records it. -1. Datadog installs the Agent on each covered instance through AWS Systems Manager, adding any missing IAM configuration automatically. -1. Datadog keeps the covered instances instrumented. Instances launched later aren't added until you update the rule. +1. You select the resources to cover, or opt in to all eligible resources. +1. Datadog resolves your selection into a set of covered resources and records it. +1. Datadog instruments each covered resource: on EC2, by installing the Agent through AWS Systems Manager; on Lambda, by adding the Datadog layers and environment variables to the function. +1. Datadog keeps the covered resources instrumented, reinstalling instrumentation that goes missing and retrying anything that failed. -You approve one CloudFormation stack, one time, during initial setup. After that, installations run automatically from Datadog, with no new CloudFormation template to launch for each installation. +You approve one CloudFormation stack, one time, during initial setup. After that, instrumentation runs automatically from Datadog, with no new CloudFormation template to launch each time. -For the full technical and security details, including the AWS resources Datadog creates, the installation mechanism, and the reconciliation model, see [How Agent installation through the AWS integration works][6]. +For the full technical and security details, including the AWS resources Datadog creates, the instrumentation mechanism, and the reconciliation model, see [How Datadog instrumentation through the AWS integration works][6]. {{< img src="integrations/amazon_web_services/aws-agent-installation-how-it-works.png" alt="Flowchart of the AWS Agent installation process, showing which steps happen in Datadog and which run inside your AWS account." style="width:70%;" >}} -## Install the Agent + + +## Install -You can start Agent installation from two entry points, depending on how much control you want over which instances are instrumented: +You can start instrumentation from two entry points, depending on how much control you want over which resources are instrumented: -- **AWS integration setup (install on all eligible instances)**: When you [set up the AWS integration][5], enable the Agent installation toggle on the [AWS integration page][7], shown alongside log and resource collection. The Agent installs on all eligible EC2 instances. -- **Fleet Automation (install on specific instances)**: Open the [AWS Install Agents page][8] at any time to select the specific EC2 instances you want. +- **AWS integration setup (instrument all eligible resources)**: When you [set up the AWS integration][5], enable the instrumentation toggle on the [AWS integration page][7], shown alongside log and resource collection, then select the workloads you want. Datadog instruments all eligible resources for those workloads. +- **Fleet Automation (instrument specific resources)**: Open the [AWS Install Agents page][8] at any time to select the specific resources you want. - + -The Agent installation toggle appears during setup: +The instrumentation toggle appears during setup, with a workload selector listing EC2 Instances, Lambda Functions, and EKS Clusters: {{< img src="integrations/amazon_web_services/aws-agent-installation-setup-toggle.png" alt="The Install the Datadog Agent step in AWS setup, with the install toggle enabled and the Hosts (EC2) workload toggle turned on." style="width:80%;" >}} + + To install from the AWS Install Agents page: -1. Opt in to all eligible instances, or select specific EC2 instances from the resource list. +1. Select the workload you want to instrument: **EC2 Instances** or **Lambda Functions**. +1. Opt in to all eligible resources, or select specific resources from the resource list. For Lambda, you can narrow the list by region, runtime, and memory size. +1. Review the preview of matching resources. Resources Datadog can't instrument appear as ineligible, with the reason. 1. Review the generated CloudFormation stack, then continue to AWS and create it. Datadog prompts you for this only once. -1. Return to Datadog. The installation proceeds automatically, and Datadog reports progress as Agents come online. +1. Return to Datadog. Instrumentation proceeds automatically, and Datadog reports progress as resources are instrumented. +## What Datadog changes on a Lambda function + +Datadog applies one `UpdateFunctionConfiguration` call per function. That call: + +- Adds the Datadog extension layer, and the Datadog tracing layer matching the function's runtime and architecture. OS-only runtimes receive the extension layer alone. +- Sets `DD_SITE` and `DD_ORG_UUID`, which the extension uses to send telemetry to your Datadog organization. +- For Node.js and Python, redirects the function handler to the Datadog handler and moves your original handler into `DD_LAMBDA_HANDLER`. +- For Java and .NET, sets `AWS_LAMBDA_EXEC_WRAPPER` to `/opt/datadog_wrapper`. + +Your existing layers, environment variables, and handler are preserved. Datadog records exactly what it changed, so uninstalling restores your original configuration. + +**No Datadog API key is written into your function.** The extension authenticates with the function's own execution role through workload identity federation, so there is no Datadog credential stored in your account for Lambda instrumentation. + +Datadog does not change your function code, memory size, timeout, VPC configuration, or any other setting. + ## Verify the installation -After the installation completes: +After instrumentation completes: -- The newly installed Agents appear in the [Infrastructure List][3] and on the host map. -- Fleet Automation lists the same Agents in the Fleet View. +- **EC2**: The newly installed Agents appear in the [Infrastructure List][3] and on the host map. Fleet Automation lists the same Agents in the Fleet View. +- **Lambda**: The instrumented functions appear on the [Serverless][11] page, and their traces appear in [APM][12]. -## Manage installed Agents +## Manage instrumented resources -Use the [AWS Install Agents page][8] in Fleet Automation to manage the Agents you've installed through the AWS integration. +Use the [AWS Install Agents page][8] in Fleet Automation to manage the resources you've instrumented through the AWS integration. From this page, you can: -- View the installed Agents and their status. -- Install the Agent on new instances in your AWS environment. -- Uninstall Agents from instances you no longer want to monitor. +- View the instrumented resources and their status. +- Instrument new resources in your AWS environment. +- Remove instrumentation from resources you no longer want to monitor. + +The rule is the source of truth. To stop coverage, update the rule. If you remove instrumentation from a covered resource yourself, Datadog restores it on the next reconciliation. Manage Agent configuration and version upgrades through [Fleet Automation][4]. + +## Uninstall -To stop coverage, update the rule. If you manually remove the Agent from a covered instance, Datadog reinstalls it on the next reconciliation. Manage Agent configuration and version upgrades through [Fleet Automation][4]. +To remove instrumentation, remove resources from a rule, edit the rule's query, or delete the rule. Deleting a rule removes instrumentation from everything the rule covered. + +- **EC2**: Datadog removes the Datadog Agent and any IAM role or instance profile it created for that instance. +- **Lambda**: Datadog removes the layers it added and restores the environment variables and handler the function had beforehand. Layers and environment variables you added yourself are left in place. ## Troubleshooting @@ -124,7 +210,25 @@ Agent installation on EC2 relies on the AWS Systems Manager (SSM) Agent, which D ### A permission or IAM error occurs -If installation can't complete because of missing permissions, Datadog shows a notification linking to the CloudFormation resource that needs the new permission. Update your existing stack to grant the [required permissions](#required-aws-permissions). You don't need to create a new stack. +If instrumentation can't complete because of missing permissions, Datadog shows a notification linking to the CloudFormation resource that needs the new permission. Update your existing stack to grant the [required permissions](#required-aws-permissions). You don't need to create a new stack. + +### A Lambda function is skipped as already instrumented + +Datadog skips any function that carries Datadog layers, a Datadog handler, or Datadog environment variables that Datadog did not apply. This prevents layer and configuration conflicts. To manage the function from the AWS integration instead, remove your existing Datadog instrumentation from it, then wait for the next reconciliation. + +Functions managed by [remote instrumentation][9] are also skipped, and reported separately from functions you instrumented yourself. A function can be managed by only one Datadog instrumentation product. + +### A Lambda function exceeds the layer limit + +AWS limits a function to five layers, and Datadog adds up to two. When a function already carries enough layers that instrumentation would exceed the limit, Datadog reports it and stops rather than retrying. Remove a layer from the function to make room, then wait for the next reconciliation. + +### A Lambda function uses a non-Datadog execution wrapper + +Java and .NET instrumentation sets `AWS_LAMBDA_EXEC_WRAPPER`. When a function already sets that variable to something other than the Datadog wrapper, Datadog skips the function rather than overwrite your wrapper. + +### A Lambda function appears as ineligible + +Datadog marks a function ineligible when it doesn't meet the [Lambda prerequisites](#aws-lambda-functions). The most common reasons are a container image package type, an unsupported runtime or architecture, a function outside the commercial `aws` partition, and Lambda@Edge functions. Lambda@Edge replicas and the functions they replicate are both excluded. ## Further reading @@ -138,3 +242,7 @@ If installation can't complete because of missing permissions, Datadog shows a n [6]: https://docs.datadoghq.com/integrations/guide/aws-agent-installation-technical-reference/ [7]: https://app.datadoghq.com/integrations/amazon-web-services [8]: https://app.datadoghq.com/fleet/install-agent/latest?platform=aws +[9]: https://docs.datadoghq.com/serverless/aws_lambda/remote_instrumentation/ +[10]: https://docs.datadoghq.com/integrations/amazon_web_services/#resource-collection +[11]: https://app.datadoghq.com/functions +[12]: https://app.datadoghq.com/apm/traces diff --git a/hugo/layouts/shortcodes/aws-agent-installation.en.md b/hugo/layouts/shortcodes/aws-agent-installation.en.md index c51db7d8ba2..b35729b658b 100644 --- a/hugo/layouts/shortcodes/aws-agent-installation.en.md +++ b/hugo/layouts/shortcodes/aws-agent-installation.en.md @@ -1,10 +1,19 @@ -Agent installation requires permissions beyond the base [AWS integration IAM policy](https://docs.datadoghq.com/integrations/amazon_web_services/#aws-iam-permissions). These permissions let Datadog: +Datadog instrumentation requires permissions beyond the base [AWS integration IAM policy](https://docs.datadoghq.com/integrations/amazon_web_services/#aws-iam-permissions). The permissions you need depend on the workloads you select. + +For **Amazon EC2 instances**, these permissions let Datadog: - Store your Datadog API key (AWS Secrets Manager). - Create and attach the IAM role used to install the Agent (IAM). - Install the Agent on your EC2 instances (AWS Systems Manager). -- Maintain the Agent's presence (Amazon EventBridge). + +For **AWS Lambda functions**, these permissions let Datadog: + +- Find your functions and read their configuration and tags (AWS Lambda). +- Add and remove the Datadog Lambda layers on a function (AWS Lambda). +- Identify Lambda@Edge functions so they can be skipped (Amazon CloudFront). + +Both workloads also use Amazon EventBridge to maintain instrumentation as your AWS resources change.
All write actions run inside your own AWS account through the role created by the CloudFormation stack. Datadog does not hold persistent write credentials to your account.
-The CloudFormation stack you approve during setup adds these permissions to your AWS integration IAM role. There is no policy to apply manually. +The CloudFormation stack you approve during setup adds these permissions to your AWS integration IAM role, scoped to the workloads you selected. There is no policy to apply manually. From 4de19459a1783d97f18c8866f01dc2dfbc5b6306 Mon Sep 17 00:00:00 2001 From: Teddy Kahwaji Date: Thu, 3 Sep 2026 12:27:24 -0400 Subject: [PATCH 2/6] Confirm frozen rule coverage, add serverless links, soften internal wording - Rule coverage is fixed at save time for Lambda as well as EC2: the UI sends is_frozen on create and update for every workload. Removes the open question and restores the workload-agnostic wording. - Link out to the existing serverless Lambda docs for extension configuration, monitoring features, and troubleshooting. - Reword implementation details that read as internal: intake mapping reconciliation, permissions boundaries, and control-plane throttling. Co-Authored-By: Claude Opus 5 (1M context) --- ...-agent-installation-technical-reference.md | 26 +++++++++---------- .../guide/aws-agent-installation.md | 20 +++++++++++--- 2 files changed, 30 insertions(+), 16 deletions(-) diff --git a/hugo/content/en/integrations/guide/aws-agent-installation-technical-reference.md b/hugo/content/en/integrations/guide/aws-agent-installation-technical-reference.md index c21c177ff76..006985b4c29 100644 --- a/hugo/content/en/integrations/guide/aws-agent-installation-technical-reference.md +++ b/hugo/content/en/integrations/guide/aws-agent-installation-technical-reference.md @@ -33,7 +33,7 @@ The CloudFormation template you launch creates the following resources one time, | IAM role | auto-named | Lets EventBridge send events to the `datadog-agent-resource-update-intake-destination` API destination | | IAM role | `datadog-eventbridge-cross-region-role` | Lets other regions forward events to your primary region | -The stack also attaches the IAM permissions for the workloads you selected to your AWS integration role. A Lambda-only selection receives no EC2 permissions, and no IAM permissions boundary. +The stack also attaches the IAM permissions for the workloads you selected to your AWS integration role. A Lambda-only selection receives no EC2 permissions. ### Created as needed, for EC2 instances @@ -67,14 +67,14 @@ Datadog does not reboot or restart your instances. The only service Datadog touc ### On AWS Lambda -Lambda instrumentation runs entirely from Datadog. Datadog does not deploy a function into your account and does not run `datadog-ci` there. +Lambda instrumentation runs entirely from Datadog. Datadog does not deploy anything into your account to instrument your functions. 1. Datadog reads the function's current configuration and tags, and checks that it meets the [Lambda prerequisites][3]. -2. Datadog checks whether the function is already instrumented. A function carrying Datadog layers, a Datadog handler, or Datadog environment variables that Datadog did not apply is skipped, as is a function managed by [remote instrumentation][4]. The two cases are reported separately. +2. Datadog checks whether the function is already instrumented. A function carrying Datadog layers, a Datadog handler, or Datadog environment variables that Datadog did not apply is skipped, as is a function managed by [remote instrumentation][4]. Datadog reports which of the two applies. 3. Datadog resolves the Datadog layer versions for the function's runtime, architecture, region, and AWS partition. Layer versions come from a pinned set that advances with Datadog's layer releases, so an installation is reproducible rather than tracking whatever is newest at that moment. -4. Datadog computes the complete desired configuration and records a checkpoint of exactly what it changes, before changing anything. -5. Datadog reconciles the intake authentication mapping for the function's execution role. See [How Lambda telemetry is authenticated](#how-lambda-telemetry-is-authenticated). -6. Datadog calls `lambda:UpdateFunctionConfiguration` once, submitting the complete layer list and environment map. The checkpoint is marked applied only after AWS reports success. +4. Datadog computes the complete desired configuration and records exactly what it is about to change, before changing anything. +5. Datadog authorizes the function's execution role to send telemetry to your Datadog organization. See [How Lambda telemetry is authenticated](#how-lambda-telemetry-is-authenticated). +6. Datadog calls `lambda:UpdateFunctionConfiguration` once, submitting the complete layer list and environment map. Datadog marks the change as applied only after AWS reports success. A Lambda update is a replace-style operation: the submitted layer list and environment map become the new configuration. Datadog therefore computes the full desired state rather than appending to it, preserving your existing layers and environment variables. The update carries the function's revision ID, so a change made in your account between Datadog's read and write causes the update to fail rather than overwrite it. @@ -126,7 +126,7 @@ The API key is stored in your own Secrets Manager, encrypted at rest. Only the s Lambda instrumentation stores no Datadog credential in your account. The Datadog extension authenticates with the function's AWS execution identity through workload identity federation, using the `DD_ORG_UUID` and `DD_SITE` values Datadog sets on the function. No Datadog API key, secret ARN, or KMS-encrypted key is written into the function's configuration. -For that authentication to succeed, Datadog maintains a mapping between the execution role's STS session pattern and your organization's managed rotation key. Datadog reconciles this mapping before submitting a function update. Mapping creation is idempotent, and Datadog accepts only an exact existing mapping rather than an overlapping one. +For that authentication to succeed, Datadog authorizes the function's execution role to send telemetry to your Datadog organization. Datadog sets up this authorization before it updates a function, and matches the execution role exactly rather than by a broader pattern. Because a single execution role is often shared across functions, Datadog creates these mappings but does not remove them on uninstall. Removing a mapping for a shared role could break another function that still depends on it. @@ -140,7 +140,7 @@ Because a single execution role is often shared across functions, Datadog create - Datadog never removes instrumentation it did not install. - Datadog tracks which resources it instrumented, so it cleans up only its own work. - On EC2, when some regions cannot be listed, Datadog skips cleanup for that pass rather than risk uninstalling in bulk. -- On Lambda, Datadog restores a function from the checkpoint it recorded, so an uninstall reverses exactly the change Datadog made. +- On Lambda, Datadog restores a function from the configuration it recorded before instrumenting it, so an uninstall reverses exactly the change Datadog made. - A failure is scoped to the individual resource. One throttled or invalid resource does not cause Datadog to reprocess resources that already succeeded. ## How Datadog maintains instrumentation @@ -152,7 +152,7 @@ Datadog continuously maintains the state you define on the covered resources: - A full reconciliation runs hourly per AWS account. Reconciliation restores instrumentation if it goes missing, retries anything that failed, and cleans up resources that no longer exist. - Change events forwarded from your account let Datadog react to covered resources within minutes, instead of waiting for the hourly pass. For EC2, these come from the CloudFormation stack's EventBridge rule. For Lambda, the `datadog-agent-resource-update-rule-lambda` rule forwards function create, configuration update, tag, and untag events. - On EC2, already-installed instances are re-verified about once per day rather than every hour, to avoid unnecessary activity. -- On Lambda, the hourly scan checks each covered function against the layer versions Datadog deploys, and does per-function work only for functions that need a change. A fleet already on current layer versions produces no per-function activity, which keeps Datadog well clear of Lambda control-plane throttling. +- On Lambda, the hourly scan checks each covered function against the layer versions Datadog deploys, and does per-function work only for functions that need a change. A fleet already on current layer versions produces no per-function activity, so Datadog makes no unnecessary calls to the Lambda API in your account. ### How Lambda functions pick up new layer versions @@ -160,11 +160,11 @@ Datadog resolves layer versions from a pinned set on every reconciliation, rathe A Lambda configuration update that is still in progress is left alone and retried shortly afterward, so Datadog does not race a change already being applied. - +### Rule coverage is fixed at save time -### Rule coverage on EC2 is fixed at save time +A rule covers the set of resources it resolved to when you saved it, and Datadog does not instrument anything outside that set. This applies to both workloads: EC2 instances launched later, and Lambda functions created later, are not picked up automatically. To cover them, update the rule, which re-resolves your query against your current fleet. -An EC2 rule covers the list of instances it resolved to when you saved it, and Datadog does not instrument anything outside that list. Instances launched later are not picked up automatically. To cover them, update the rule, which re-resolves your query against your current fleet. +Change events are what keeps the covered set correct, not what expands it. A forwarded event causes Datadog to re-examine a resource the rule already covers. ### What happens when you edit a rule @@ -187,7 +187,7 @@ When someone removes instrumentation from a covered resource by hand, the next r To uninstall, remove resources from a rule, edit the rule's query, or delete the rule. - **EC2**: Uninstalling removes the Datadog Agent, the `/etc/datadog-agent` and `/opt/datadog-agent` directories on Linux (or performs an MSI uninstall on Windows), and any IAM role or instance profile Datadog created for that instance. -- **Lambda**: Uninstalling removes the Datadog layers Datadog added and restores the environment variables and handler recorded in the checkpoint. Datadog validates the checkpoint against the function's current configuration first, so it does not remove a layer or variable it did not add. The intake authentication mapping for the execution role is left in place, because the role may be shared with other functions. +- **Lambda**: Uninstalling removes the Datadog layers Datadog added and restores the environment variables and handler the function had beforehand. Datadog checks that record against the function's current configuration first, so it does not remove a layer or variable it did not add. The telemetry authorization for the execution role is left in place, because the role may be shared with other functions. ## Further reading diff --git a/hugo/content/en/integrations/guide/aws-agent-installation.md b/hugo/content/en/integrations/guide/aws-agent-installation.md index 6465ae315ff..8664eeaff23 100644 --- a/hugo/content/en/integrations/guide/aws-agent-installation.md +++ b/hugo/content/en/integrations/guide/aws-agent-installation.md @@ -21,9 +21,18 @@ further_reading: - link: "https://docs.datadoghq.com/agent/configuration/" tag: "Documentation" text: "Agent Configuration" +- link: "https://docs.datadoghq.com/serverless/aws_lambda/" + tag: "Documentation" + text: "Serverless Monitoring for AWS Lambda" +- link: "https://docs.datadoghq.com/serverless/aws_lambda/configuration/" + tag: "Documentation" + text: "Configure Serverless Monitoring for AWS Lambda" - link: "https://docs.datadoghq.com/serverless/aws_lambda/instrumentation/" tag: "Documentation" text: "Instrumenting AWS Lambda" +- link: "https://docs.datadoghq.com/serverless/aws_lambda/troubleshooting/" + tag: "Documentation" + text: "Troubleshoot AWS Lambda Monitoring" --- ## Overview @@ -124,7 +133,7 @@ Instrumentation is based on an **instrumentation rule**: an AWS account paired w 1. You select the resources to cover, or opt in to all eligible resources. 1. Datadog resolves your selection into a set of covered resources and records it. 1. Datadog instruments each covered resource: on EC2, by installing the Agent through AWS Systems Manager; on Lambda, by adding the Datadog layers and environment variables to the function. -1. Datadog keeps the covered resources instrumented, reinstalling instrumentation that goes missing and retrying anything that failed. +1. Datadog keeps the covered resources instrumented, reinstalling instrumentation that goes missing and retrying anything that failed. Resources created later aren't added until you update the rule. You approve one CloudFormation stack, one time, during initial setup. After that, instrumentation runs automatically from Datadog, with no new CloudFormation template to launch each time. @@ -174,12 +183,14 @@ Your existing layers, environment variables, and handler are preserved. Datadog Datadog does not change your function code, memory size, timeout, VPC configuration, or any other setting. +Datadog preserves environment variables you set yourself, so you can tune what the extension collects with the standard Datadog environment variables. For the full list, see [Configure Serverless Monitoring for AWS Lambda][14]. For what instrumentation collects and the Lambda monitoring features it enables, see [Serverless Monitoring for AWS Lambda][13]. + ## Verify the installation After instrumentation completes: - **EC2**: The newly installed Agents appear in the [Infrastructure List][3] and on the host map. Fleet Automation lists the same Agents in the Fleet View. -- **Lambda**: The instrumented functions appear on the [Serverless][11] page, and their traces appear in [APM][12]. +- **Lambda**: The instrumented functions appear on the [Serverless][11] page, and their traces appear in [APM][12]. If a function is instrumented but its telemetry doesn't arrive, see [Troubleshoot AWS Lambda Monitoring][15]. @@ -216,7 +227,7 @@ If instrumentation can't complete because of missing permissions, Datadog shows Datadog skips any function that carries Datadog layers, a Datadog handler, or Datadog environment variables that Datadog did not apply. This prevents layer and configuration conflicts. To manage the function from the AWS integration instead, remove your existing Datadog instrumentation from it, then wait for the next reconciliation. -Functions managed by [remote instrumentation][9] are also skipped, and reported separately from functions you instrumented yourself. A function can be managed by only one Datadog instrumentation product. +Functions managed by [remote instrumentation][9] are also skipped, and Datadog tells you which of the two applies. A function can be managed by only one Datadog instrumentation product. ### A Lambda function exceeds the layer limit @@ -246,3 +257,6 @@ Datadog marks a function ineligible when it doesn't meet the [Lambda prerequisit [10]: https://docs.datadoghq.com/integrations/amazon_web_services/#resource-collection [11]: https://app.datadoghq.com/functions [12]: https://app.datadoghq.com/apm/traces +[13]: https://docs.datadoghq.com/serverless/aws_lambda/ +[14]: https://docs.datadoghq.com/serverless/aws_lambda/configuration/ +[15]: https://docs.datadoghq.com/serverless/aws_lambda/troubleshooting/ From cf795d4ae85bb8588ebc80d4ee766486fbc04a86 Mon Sep 17 00:00:00 2001 From: Teddy Kahwaji Date: Thu, 3 Sep 2026 12:42:33 -0400 Subject: [PATCH 3/6] Link Workload Identity Federation docs from the Lambda sections Lambda instrumentation authenticates through Workload Identity Federation, but neither page linked the WIF documentation. Adds the link inline and to further reading on both pages, and states that Datadog sets up the authorization so there is nothing for the customer to configure. Flags a pre-publish dependency: WIF for intake is documented as Enterprise-only, and Lambda instrumentation needs it available to the orgs this feature targets. Co-Authored-By: Claude Opus 5 (1M context) --- .../guide/aws-agent-installation-technical-reference.md | 8 +++++++- .../en/integrations/guide/aws-agent-installation.md | 6 +++++- 2 files changed, 12 insertions(+), 2 deletions(-) diff --git a/hugo/content/en/integrations/guide/aws-agent-installation-technical-reference.md b/hugo/content/en/integrations/guide/aws-agent-installation-technical-reference.md index 006985b4c29..70d69b6d2fe 100644 --- a/hugo/content/en/integrations/guide/aws-agent-installation-technical-reference.md +++ b/hugo/content/en/integrations/guide/aws-agent-installation-technical-reference.md @@ -12,6 +12,9 @@ further_reading: - link: "https://docs.datadoghq.com/agent/fleet_automation/" tag: "Documentation" text: "Fleet Automation" +- link: "https://docs.datadoghq.com/account_management/workload_identity_federation/" + tag: "Documentation" + text: "Workload Identity Federation" --- This page explains how Datadog instruments and maintains your AWS workloads through the AWS integration. For setup instructions and the permissions Datadog requires, see [Install Datadog instrumentation through the AWS Integration][1]. @@ -124,7 +127,9 @@ The API key is stored in your own Secrets Manager, encrypted at rest. Only the s ### How Lambda telemetry is authenticated -Lambda instrumentation stores no Datadog credential in your account. The Datadog extension authenticates with the function's AWS execution identity through workload identity federation, using the `DD_ORG_UUID` and `DD_SITE` values Datadog sets on the function. No Datadog API key, secret ARN, or KMS-encrypted key is written into the function's configuration. +Lambda instrumentation stores no Datadog credential in your account. The Datadog extension authenticates with the function's AWS execution identity through [Workload Identity Federation][5], using the `DD_ORG_UUID` and `DD_SITE` values Datadog sets on the function. No Datadog API key, secret ARN, or KMS-encrypted key is written into the function's configuration. + + For that authentication to succeed, Datadog authorizes the function's execution role to send telemetry to your Datadog organization. Datadog sets up this authorization before it updates a function, and matches the execution role exactly rather than by a broader pattern. @@ -197,3 +202,4 @@ To uninstall, remove resources from a rule, edit the rule's query, or delete the [2]: https://docs.datadoghq.com/integrations/guide/aws-agent-installation/#prerequisites [3]: https://docs.datadoghq.com/integrations/guide/aws-agent-installation/#aws-lambda-functions [4]: https://docs.datadoghq.com/serverless/aws_lambda/remote_instrumentation/ +[5]: https://docs.datadoghq.com/account_management/workload_identity_federation/ diff --git a/hugo/content/en/integrations/guide/aws-agent-installation.md b/hugo/content/en/integrations/guide/aws-agent-installation.md index 8664eeaff23..eb29aabad06 100644 --- a/hugo/content/en/integrations/guide/aws-agent-installation.md +++ b/hugo/content/en/integrations/guide/aws-agent-installation.md @@ -33,6 +33,9 @@ further_reading: - link: "https://docs.datadoghq.com/serverless/aws_lambda/troubleshooting/" tag: "Documentation" text: "Troubleshoot AWS Lambda Monitoring" +- link: "https://docs.datadoghq.com/account_management/workload_identity_federation/" + tag: "Documentation" + text: "Workload Identity Federation" --- ## Overview @@ -179,7 +182,7 @@ Datadog applies one `UpdateFunctionConfiguration` call per function. That call: Your existing layers, environment variables, and handler are preserved. Datadog records exactly what it changed, so uninstalling restores your original configuration. -**No Datadog API key is written into your function.** The extension authenticates with the function's own execution role through workload identity federation, so there is no Datadog credential stored in your account for Lambda instrumentation. +**No Datadog API key is written into your function.** The extension authenticates with the function's own execution role through [Workload Identity Federation][16], so there is no Datadog credential stored in your account for Lambda instrumentation. Datadog sets up that authorization for you; there is nothing to configure. Datadog does not change your function code, memory size, timeout, VPC configuration, or any other setting. @@ -260,3 +263,4 @@ Datadog marks a function ineligible when it doesn't meet the [Lambda prerequisit [13]: https://docs.datadoghq.com/serverless/aws_lambda/ [14]: https://docs.datadoghq.com/serverless/aws_lambda/configuration/ [15]: https://docs.datadoghq.com/serverless/aws_lambda/troubleshooting/ +[16]: https://docs.datadoghq.com/account_management/workload_identity_federation/ From bb751b57456507be01fab78de3b9cdcb8375b289 Mon Sep 17 00:00:00 2001 From: Teddy Kahwaji Date: Thu, 3 Sep 2026 12:50:17 -0400 Subject: [PATCH 4/6] Assume WIF is generally available and name it in the partition note Drops the pre-publish caveat about Enterprise-only gating for Workload Identity Federation; these pages stay gated with private: true until the rollout, by which point the gate is lifted. Also names Workload Identity Federation in the AWS partition prerequisite, which described it as "cloud-based authentication", and links it there. Co-Authored-By: Claude Opus 5 (1M context) --- .../guide/aws-agent-installation-technical-reference.md | 2 -- hugo/content/en/integrations/guide/aws-agent-installation.md | 2 +- 2 files changed, 1 insertion(+), 3 deletions(-) diff --git a/hugo/content/en/integrations/guide/aws-agent-installation-technical-reference.md b/hugo/content/en/integrations/guide/aws-agent-installation-technical-reference.md index 70d69b6d2fe..8edcb71da7f 100644 --- a/hugo/content/en/integrations/guide/aws-agent-installation-technical-reference.md +++ b/hugo/content/en/integrations/guide/aws-agent-installation-technical-reference.md @@ -129,8 +129,6 @@ The API key is stored in your own Secrets Manager, encrypted at rest. Only the s Lambda instrumentation stores no Datadog credential in your account. The Datadog extension authenticates with the function's AWS execution identity through [Workload Identity Federation][5], using the `DD_ORG_UUID` and `DD_SITE` values Datadog sets on the function. No Datadog API key, secret ARN, or KMS-encrypted key is written into the function's configuration. - - For that authentication to succeed, Datadog authorizes the function's execution role to send telemetry to your Datadog organization. Datadog sets up this authorization before it updates a function, and matches the execution role exactly rather than by a broader pattern. Because a single execution role is often shared across functions, Datadog creates these mappings but does not remove them on uninstall. Removing a mapping for a shared role could break another function that still depends on it. diff --git a/hugo/content/en/integrations/guide/aws-agent-installation.md b/hugo/content/en/integrations/guide/aws-agent-installation.md index eb29aabad06..9f851682534 100644 --- a/hugo/content/en/integrations/guide/aws-agent-installation.md +++ b/hugo/content/en/integrations/guide/aws-agent-installation.md @@ -70,7 +70,7 @@ For all workloads, confirm the following: ### AWS Lambda functions - **Resource collection**: [Resource collection][10] must be enabled on the AWS integration. Datadog uses it to list your functions and preview which ones a rule matches. -- **AWS partition**: The commercial `aws` partition only. Functions in AWS GovCloud or the AWS China partitions are not supported, because Lambda instrumentation authenticates through cloud-based authentication, which those partitions don't support. +- **AWS partition**: The commercial `aws` partition only. Functions in AWS GovCloud or the AWS China partitions are not supported, because Lambda instrumentation authenticates through [Workload Identity Federation][16], which those partitions don't support. - **Package type**: Zip functions only. Container image functions are not supported, because Datadog instrumentation is distributed as Lambda layers, which container image functions can't use. - **Architecture**: `x86_64` or `arm64`. A function must report a single architecture. - **Layer count**: AWS limits a function to five layers. Datadog adds up to two, so a function that already carries four or more layers is not eligible. From 746639d8ce595045788c9b9d206d88d905185656 Mon Sep 17 00:00:00 2001 From: Teddy Kahwaji Date: Fri, 4 Sep 2026 12:13:56 -0400 Subject: [PATCH 5/6] Disambiguate AWS integration instrumentation from remote instrumentation Add a comparison section to the setup guide covering where the work runs, how functions are selected, layer version control, credentials, and removal, plus a pointer to it from the technical reference. Co-Authored-By: Claude Opus 5 --- ...-agent-installation-technical-reference.md | 3 +++ .../guide/aws-agent-installation.md | 25 ++++++++++++++++++- 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/hugo/content/en/integrations/guide/aws-agent-installation-technical-reference.md b/hugo/content/en/integrations/guide/aws-agent-installation-technical-reference.md index 8edcb71da7f..d02af0bc65a 100644 --- a/hugo/content/en/integrations/guide/aws-agent-installation-technical-reference.md +++ b/hugo/content/en/integrations/guide/aws-agent-installation-technical-reference.md @@ -21,6 +21,8 @@ This page explains how Datadog instruments and maintains your AWS workloads thro This page covers Amazon EC2 instances and AWS Lambda functions. Amazon EKS is not supported. +Datadog also offers [remote instrumentation][4] for Lambda functions, which deploys an instrumenter function into your own account rather than making the changes from Datadog. For a comparison of the two, see [Choose between this guide and remote instrumentation][6]. + ## AWS resources that Datadog creates ### Created once, by the CloudFormation stack @@ -201,3 +203,4 @@ To uninstall, remove resources from a rule, edit the rule's query, or delete the [3]: https://docs.datadoghq.com/integrations/guide/aws-agent-installation/#aws-lambda-functions [4]: https://docs.datadoghq.com/serverless/aws_lambda/remote_instrumentation/ [5]: https://docs.datadoghq.com/account_management/workload_identity_federation/ +[6]: https://docs.datadoghq.com/integrations/guide/aws-agent-installation/#choose-between-this-guide-and-remote-instrumentation diff --git a/hugo/content/en/integrations/guide/aws-agent-installation.md b/hugo/content/en/integrations/guide/aws-agent-installation.md index 9f851682534..190fee76948 100644 --- a/hugo/content/en/integrations/guide/aws-agent-installation.md +++ b/hugo/content/en/integrations/guide/aws-agent-installation.md @@ -51,10 +51,33 @@ You can instrument your AWS workloads directly from Datadog, without connecting | Amazon EC2 instances | The Datadog Agent | | AWS Lambda functions | The Datadog Lambda extension, and the Datadog tracing layer matching the function's runtime | -Amazon EKS is not supported. For Lambda functions, this guide covers instrumentation managed from Datadog. To instrument functions with a Lambda function deployed in your own account instead, see [Remote instrumentation for AWS Lambda][9]. +Amazon EKS is not supported. For Lambda functions, Datadog offers a second product, remote instrumentation, which runs an instrumenter function in your own account. See [Choose between this guide and remote instrumentation](#choose-between-this-guide-and-remote-instrumentation).
A Lambda function can be managed by only one Datadog instrumentation product. Datadog skips any function that remote instrumentation already manages, and skips any function you instrumented yourself.
+## Choose between this guide and remote instrumentation + +Datadog offers two ways to add instrumentation to Lambda functions without redeploying them yourself: + +- **Instrumentation through the AWS integration**, covered by this guide, is managed entirely from Datadog. Datadog updates your functions with the IAM role created by the CloudFormation stack, and deploys no compute into your account. +- **[Remote instrumentation][9]** deploys a Datadog instrumenter function, `datadog-remote-instrumenter`, into your own account. That function applies the instrumentation and keeps it in place. + +Both add the same Datadog Lambda extension and tracing layers, and both restore instrumentation that is changed outside of Datadog. They differ in where the work runs, how functions are selected, and what you install. + +| | Instrumentation through the AWS integration | Remote instrumentation | +|---|---|---| +| Workloads | Amazon EC2 instances and AWS Lambda functions | AWS Lambda functions | +| What Datadog deploys in your account | No compute. Datadog calls the AWS APIs with the IAM role created by the CloudFormation stack | An instrumenter Lambda function, a CloudTrail, and supporting resources | +| Scope of setup | One CloudFormation stack per AWS account | One CloudFormation stack per account and region | +| Selecting functions | You choose the functions a rule covers, and review the matched set before saving | You write targeting rules on function names and tags, with logical operators | +| Functions created later | Not instrumented. The covered set is fixed when you save the rule | Instrumented automatically when they match your targeting rules | +| Layer versions | Datadog selects and updates them | You set them, and they stay fixed until you change them | +| Datadog credentials in your account | None for Lambda. Instrumented functions authenticate with [Workload Identity Federation][16] | A Datadog API key with Remote Configuration enabled | +| Datadog permissions | Hosts Read and Agent Install | Serverless AWS Instrumentation Read and Write | +| Removing instrumentation | Uninstall from Datadog | Delete the CloudFormation stack in that region | + +Use instrumentation through the AWS integration when you want one place to instrument both EC2 instances and Lambda functions, want nothing extra deployed in your account, or want to avoid storing a Datadog API key there. Use remote instrumentation when you need functions created later to be instrumented as they appear, or when you want to control the layer versions applied to your functions. + ## Prerequisites For all workloads, confirm the following: From a76bc032f2ed793520944466cdc3b0384d69bdb1 Mon Sep 17 00:00:00 2001 From: Teddy Kahwaji Date: Fri, 4 Sep 2026 13:57:07 -0400 Subject: [PATCH 6/6] Label the comparison table's first column and tighten its claims The header row began with an empty cell, which Hugo drops, shifting the header labels one column left and leaving the third column unlabeled. Also scope the authentication row to the function, since the stack's EventBridge connection does hold Datadog keys in the account, and keep the "what runs in your account" row to compute only. Co-Authored-By: Claude Opus 5 --- .../en/integrations/guide/aws-agent-installation.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/hugo/content/en/integrations/guide/aws-agent-installation.md b/hugo/content/en/integrations/guide/aws-agent-installation.md index 190fee76948..560190a4f61 100644 --- a/hugo/content/en/integrations/guide/aws-agent-installation.md +++ b/hugo/content/en/integrations/guide/aws-agent-installation.md @@ -64,19 +64,23 @@ Datadog offers two ways to add instrumentation to Lambda functions without redep Both add the same Datadog Lambda extension and tracing layers, and both restore instrumentation that is changed outside of Datadog. They differ in where the work runs, how functions are selected, and what you install. -| | Instrumentation through the AWS integration | Remote instrumentation | +| Aspect | Instrumentation through the AWS integration | Remote instrumentation | |---|---|---| | Workloads | Amazon EC2 instances and AWS Lambda functions | AWS Lambda functions | -| What Datadog deploys in your account | No compute. Datadog calls the AWS APIs with the IAM role created by the CloudFormation stack | An instrumenter Lambda function, a CloudTrail, and supporting resources | +| What runs in your account | No Datadog compute. Datadog calls the AWS APIs with the IAM role created by the CloudFormation stack | The instrumenter Lambda function | | Scope of setup | One CloudFormation stack per AWS account | One CloudFormation stack per account and region | | Selecting functions | You choose the functions a rule covers, and review the matched set before saving | You write targeting rules on function names and tags, with logical operators | | Functions created later | Not instrumented. The covered set is fixed when you save the rule | Instrumented automatically when they match your targeting rules | | Layer versions | Datadog selects and updates them | You set them, and they stay fixed until you change them | -| Datadog credentials in your account | None for Lambda. Instrumented functions authenticate with [Workload Identity Federation][16] | A Datadog API key with Remote Configuration enabled | +| How instrumented functions authenticate | [Workload Identity Federation][16], with no Datadog API key on the function | A Datadog API key with Remote Configuration enabled | | Datadog permissions | Hosts Read and Agent Install | Serverless AWS Instrumentation Read and Write | | Removing instrumentation | Uninstall from Datadog | Delete the CloudFormation stack in that region | -Use instrumentation through the AWS integration when you want one place to instrument both EC2 instances and Lambda functions, want nothing extra deployed in your account, or want to avoid storing a Datadog API key there. Use remote instrumentation when you need functions created later to be instrumented as they appear, or when you want to control the layer versions applied to your functions. +Both products deploy a CloudFormation stack in your account. The stack for remote instrumentation also creates a CloudTrail and supporting resources. For what the stack in this guide creates, including the EventBridge resources that send change events to Datadog, see [How Datadog instrumentation through the AWS integration works][6]. + +Use instrumentation through the AWS integration to instrument both EC2 instances and Lambda functions from one place. It also suits you if you want no additional compute running in your account. + +Use remote instrumentation when functions created later must be instrumented as they appear. It also suits you if you want to control the layer versions applied to your functions. ## Prerequisites