diff --git a/content/en/tracing/trace_collection/_index.md b/content/en/tracing/trace_collection/_index.md index 1ed02680a2f..20fe1bbcea5 100644 --- a/content/en/tracing/trace_collection/_index.md +++ b/content/en/tracing/trace_collection/_index.md @@ -1,6 +1,6 @@ --- -title: Application Instrumentation -description: "Get Started with Datadog APM" +title: Set Up APM +description: "Set up Datadog APM to collect traces from your applications. Choose Single Step Instrumentation for automatic setup on Kubernetes, Linux, Docker, or Windows, or use manually managed SDKs for full configuration control." aliases: - /tracing/setup - /tracing/send_traces/ @@ -31,149 +31,44 @@ algolia: --- ## Overview -Application {{< tooltip glossary="instrumentation" >}} with Datadog APM involves: -1. **SDK setup**: Adding a Datadog SDK to your application. -2. **Span creation**: Capturing observability data as {{< tooltip glossary="span" >}}s. +Set up Datadog APM to collect traces from your applications. [Single Step Instrumentation][1] (SSI) is the recommended approach for most users. It automatically installs and configures Datadog SDKs with no code changes required. - Spans are automatically generated by default as soon as the SDK is loaded. This is known as **auto-instrumentation** and provides sufficient visibility for most users. If you need more control, you can optionally add custom spans. - -**Note**: These steps assume you have a [Datadog Agent][5] installed and configured to receive traces. - -{{< img src="tracing/visualization/troubleshooting_pipeline.png" alt="The APM pipeline">}} - -## Getting started +## Get APM running
Prefer vendor-neutral instrumentation? See the OpenTelemetry documentation for using OpenTelemetry with Datadog.
-### Single step instrumentation (recommended) - -[Single Step Instrumentation][1] (SSI) automatically installs and configures Datadog SDKs with a single command. Auto-instrumentation then immediately begins capturing traces from your supported frameworks and libraries, with no code changes required. +SSI automatically installs and loads Datadog SDKs into your application processes at runtime with a single command. Pick your platform to get started: {{< whatsnext desc=" " >}} - {{< nextlink href="/tracing/trace_collection/single-step-apm/" >}}Get started with Single Step Instrumentation{{< /nextlink >}} + {{< nextlink href="/tracing/trace_collection/single-step-apm/kubernetes" >}}Kubernetes{{< /nextlink >}} + {{< nextlink href="/tracing/trace_collection/single-step-apm/linux" >}}Linux{{< /nextlink >}} + {{< nextlink href="/tracing/trace_collection/single-step-apm/docker" >}}Docker{{< /nextlink >}} + {{< nextlink href="/tracing/trace_collection/single-step-apm/windows" >}}Windows{{< /nextlink >}} {{< /whatsnext >}} -### Manual setup and custom spans +## Need more control? -As your observability needs grow, you can add more control and customization: - -**For full SDK configuration control:** Use [manually managed Datadog SDKs][2] if you need granular control over SDK behavior and configuration. +If you need full control over SDK configuration and installation, use [manually managed Datadog SDKs][2]. This approach requires adding the SDK to your application code or build process. {{< whatsnext desc=" " >}} - {{< nextlink href="/tracing/trace_collection/dd_libraries/" >}}Use manually managed Datadog SDKs{{< /nextlink >}} + {{< nextlink href="/tracing/trace_collection/dd_libraries/" >}}Set up manually managed SDKs{{< /nextlink >}} {{< /whatsnext >}} -**For custom spans without code changes:** Use [Dynamic Instrumentation][4] to create custom spans from the Datadog UI without redeploying your application. +## Add custom spans -{{< whatsnext desc=" " >}} - {{< nextlink href="/tracing/trace_collection/dynamic_instrumentation/" >}}Add custom spans with Dynamic Instrumentation{{< /nextlink >}} -{{< /whatsnext >}} +After you have APM running, you can add custom spans for visibility into application-specific code paths: -**For custom spans in code:** Add [code-based custom instrumentation][3] to instrument custom business logic or add application-specific metadata to spans. +- **Without code changes**: Use [Dynamic Instrumentation][4] to create custom spans from the Datadog UI without redeploying your application. +- **In code**: Add [code-based custom instrumentation][3] to instrument custom business logic or add application-specific metadata to spans. {{< whatsnext desc=" " >}} + {{< nextlink href="/tracing/trace_collection/dynamic_instrumentation/" >}}Add custom spans with Dynamic Instrumentation{{< /nextlink >}} {{< nextlink href="/tracing/trace_collection/custom_instrumentation/" >}}Add custom spans with code-based instrumentation{{< /nextlink >}} {{< /whatsnext >}} -These options can be combined. For example, you can start with Single Step Instrumentation and add code-based custom instrumentation for specific spans, or use manually managed SDKs with Dynamic Instrumentation for no-deploy span additions. - -## Detailed comparison - -### SDK setup - -Single Step Instrumentation is the recommended starting point for most users. If you need more control over SDK configuration, you can use manually managed SDKs instead: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Single Step Instrumentation (recommended)Manually managed SDKs
How it worksDatadog automatically installs and loads SDKs into your application processes, at runtime, with a single command.You install and configure SDKs directly in your application code or build process.
Code changes?NoYes
Setup complexityLow - minimal configuration neededMedium - requires environment and build configuration
Configuration controlStandard defaults with optional overridesFull control through environment variables and code
When to useStart here for fast, consistent instrumentation across services without code changes.Progress to this when you need granular control over SDK behavior and configuration.
- -### Span customization - -Auto-instrumentation automatically creates spans for supported frameworks and libraries, providing essential observability with no additional work. When you need visibility into custom code paths or want to enrich traces with application-specific data, you can add custom spans using either Dynamic Instrumentation or code-based custom instrumentation: - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Dynamic InstrumentationCode-based custom instrumentation
How it worksConfigure instrumentation rules in the Datadog UI; rules are applied at runtime.Add explicit tracing API calls in your application code.
Code changes?NoYes
Deployment requiredNoYes (to add or modify spans)
When to useAdd custom spans without code changes or redeployments.Progress to this when you need complex instrumentation logic or want spans permanently defined in code.
- -## APM setup tutorials - -The following tutorials guide you through setting up distributed tracing for a sample application on various infrastructure scenarios, with both automatic and custom instrumentation: - -{{< whatsnext desc="Choose your language and environment:" >}} - {{< nextlink href="tracing/guide/tutorial-enable-python-host" >}} Enabling Tracing on a Python Application on the Same Host as Datadog Agent{{< /nextlink >}} - {{< nextlink href="tracing/guide/tutorial-enable-python-containers" >}} Enabling Tracing on a Python Application and Datadog Agent in Containers{{< /nextlink >}} - {{< nextlink href="tracing/guide/tutorial-enable-python-container-agent-host" >}} Enabling Tracing for a Python Application in a Container and an Agent on a Host{{< /nextlink >}} - {{< nextlink href="tracing/guide/tutorial-enable-java-host" >}} Enabling Tracing on a Java Application on the Same Host as Datadog Agent{{< /nextlink >}} - {{< nextlink href="tracing/guide/tutorial-enable-java-containers" >}} Enabling Tracing on a Java Application and Datadog Agent in Containers{{< /nextlink >}} - {{< nextlink href="tracing/guide/tutorial-enable-java-container-agent-host" >}} Enabling Tracing for a Java Application in a Container and an Agent on a Host{{< /nextlink >}} - {{< nextlink href="tracing/guide/tutorial-enable-java-gke" >}} Enabling Tracing for a Java Application on GKE{{< /nextlink >}} - {{< nextlink href="tracing/guide/tutorial-enable-java-aws-eks" >}} Enabling Tracing for a Java Application on Amazon EKS{{< /nextlink >}} - {{< nextlink href="tracing/guide/tutorial-enable-java-aws-ecs-ec2" >}} Enabling Tracing for a Java Application in Amazon ECS with EC2{{< /nextlink >}} - {{< nextlink href="tracing/guide/tutorial-enable-java-aws-ecs-fargate" >}} Enabling Tracing for a Java Application in Amazon ECS with Fargate{{< /nextlink >}} - {{< nextlink href="tracing/guide/tutorial-enable-java-admission-controller" >}} Enabling Tracing for a Java Application with the Admission Controller{{< /nextlink >}} - {{< nextlink href="tracing/guide/tutorial-enable-go-host" >}} Enabling Tracing on a Go Application on the Same Host as Datadog Agent{{< /nextlink >}} - {{< nextlink href="tracing/guide/tutorial-enable-go-containers" >}} Enabling Tracing on a Go Application and Datadog Agent in Containers{{< /nextlink >}} - {{< nextlink href="tracing/guide/tutorial-enable-go-aws-ecs-ec2" >}} Enabling Tracing for a Go Application in Amazon ECS with EC2{{< /nextlink >}} - {{< nextlink href="tracing/guide/tutorial-enable-go-aws-ecs-fargate" >}} Enabling Tracing for a Go Application in Amazon ECS with Fargate{{< /nextlink >}} - -{{< /whatsnext >}} - ## Further reading {{< partial name="whats-next/whats-next.html" >}} @@ -182,5 +77,3 @@ The following tutorials guide you through setting up distributed tracing for a s [2]: /tracing/trace_collection/dd_libraries/ [3]: /tracing/trace_collection/custom_instrumentation/ [4]: /tracing/trace_collection/dynamic_instrumentation/ -[5]: /agent/ - diff --git a/content/en/tracing/trace_collection/single-step-apm/_index.md b/content/en/tracing/trace_collection/single-step-apm/_index.md index d91068ebb79..f455ab0da4c 100644 --- a/content/en/tracing/trace_collection/single-step-apm/_index.md +++ b/content/en/tracing/trace_collection/single-step-apm/_index.md @@ -1,5 +1,6 @@ --- title: Single Step APM Instrumentation +description: "Automatically instrument applications with Datadog APM using Single Step Instrumentation (SSI). Install the Datadog Agent and enable tracing with no code changes on Kubernetes, Linux, Docker, or Windows." aliases: - /tracing/trace_collection/admission_controller/ - /tracing/trace_collection/library_injection_local/ @@ -11,7 +12,7 @@ further_reading: - link: /tracing/guide/injectors tag: Documentation text: Understanding injector behavior with Single Step Instrumentation - - link: /tracing/trace_collection/automatic_instrumentation/single-step-apm/troubleshooting/ + - link: /tracing/trace_collection/single-step-apm/troubleshooting/ tag: Documentation text: "Troubleshooting Single Step APM" - link: https://learn.datadoghq.com/courses/troubleshooting-apm-instrumentation-on-a-host @@ -29,21 +30,21 @@ further_reading: --- ## Overview -Single Step Instrumentation (SSI) automatically installs the Datadog SDKs with no additional configuration required, reducing onboarding time from days to minutes. +Single Step Instrumentation (SSI) automatically installs the Datadog SDKs into your applications with no code changes or additional configuration required. -To learn more about how it works, see the [injector guide for Single Step Instrumentation][8]. +To learn more about how it works, see the [injector guide for Single Step Instrumentation][2]. ## Prerequisites -1. Remove any custom instrumentation code from your application and restart it. SSI is automatically disabled if custom instrumentation is detected. -1. Confirm environment compatibility by reviewing the [SSI compatibility guide][18] for supported languages, operating systems, and architectures. +1. Check whether the application already loads a tracing SDK. Remove or disable only conflicting tracer initialization before enabling SSI, then restart the application. You can keep custom spans and non-tracing OpenTelemetry dependencies. +1. Confirm environment compatibility by reviewing the [SSI compatibility guide][5] for supported languages, operating systems, and architectures. ## Instrument SDKs across applications When you [install or update the Datadog Agent][1] with {{< ui >}}APM Instrumentation{{< /ui >}} enabled, the Agent instruments your applications by loading the Datadog SDK into supported processes. This enables distributed tracing by capturing and sending trace data from your services without requiring code changes. After instrumentation, you can optionally: -- [configure Unified Service Tags (USTs)][14] +- [configure Unified Service Tags (USTs)][3] - enable additional SDK-dependent products and features, such as Continuous Profiler or Application Security Monitoring Click on one of the following tiles to learn how to set up SSI for your deployment type: @@ -59,27 +60,14 @@ Click on one of the following tiles to learn how to set up SSI for your deployme ## Troubleshooting -If you encounter problems enabling APM with SSI, see the [SSI troubleshooting guide][15]. +If you encounter problems enabling APM with SSI, see the [SSI troubleshooting guide][4]. ## Further reading {{< partial name="whats-next/whats-next.html" >}} [1]: https://app.datadoghq.com/account/settings/agent/latest -[2]: /tracing/metrics/runtime_metrics/ -[3]: /internal_developer_portal/catalog/ -[4]: /tracing/glossary/#instrumentation -[5]: /containers/cluster_agent/admission_controller/ -[6]: /tracing/trace_collection/automatic_instrumentation/single-step-apm/compatibility -[7]: /tracing/trace_collection/custom_instrumentation/ -[8]: /tracing/guide/injectors -[9]: /tracing/trace_collection/automatic_instrumentation/single-step-apm/kubernetes/?tab=installingwithdatadogoperator#configure-instrumentation-for-namespaces-and-pods -[10]: /tracing/trace_collection/library_config/ -[11]: /tracing/metrics/runtime_metrics/ -[12]: /internal_developer_portal/catalog/ -[13]: /tracing/glossary/#instrumentation -[14]: /getting_started/tagging/unified_service_tagging -[15]: /tracing/trace_collection/automatic_instrumentation/single-step-apm/troubleshooting -[16]: /tracing/trace_collection/custom_instrumentation/ -[17]: /tracing/trace_collection/library_config/application_monitoring_yaml/ -[18]: /tracing/trace_collection/automatic_instrumentation/single-step-apm/compatibility/ +[2]: /tracing/guide/injectors +[3]: /getting_started/tagging/unified_service_tagging +[4]: /tracing/trace_collection/single-step-apm/troubleshooting +[5]: /tracing/trace_collection/single-step-apm/compatibility/ diff --git a/content/en/tracing/trace_collection/single-step-apm/compatibility.md b/content/en/tracing/trace_collection/single-step-apm/compatibility.md index 0cfac419d84..72b8e4e62e4 100644 --- a/content/en/tracing/trace_collection/single-step-apm/compatibility.md +++ b/content/en/tracing/trace_collection/single-step-apm/compatibility.md @@ -1,9 +1,10 @@ --- title: Compatibility +description: "Review the supported operating systems, deployment environments, language runtimes, and limitations for Datadog Single Step Instrumentation." aliases: - /tracing/trace_collection/automatic_instrumentation/single-step-apm/compatibility/ further_reading: -- link: /tracing/trace_collection/automatic_instrumentation/single-step-apm/ +- link: /tracing/trace_collection/single-step-apm/ tag: "Documentation" text: "Single Step APM Instrumentation" --- @@ -24,19 +25,21 @@ Select your environment to see compatibility requirements and limitations: ### Compatibility - **Status**: GA -- **Supported operating systems**: See [Linux distributions reference](#linux-distributions-reference) +- **Supported operating systems**: See [Linux distributions reference][1] - **Supported architectures**: x86_64, arm64 ### Requirements - Datadog Agent with APM Instrumentation enabled -- A [supported Linux distribution](#linux-distributions-reference) +- A [supported Linux distribution][1] ### Limitations - **SELinux**: Hardened SELinux environments are not supported. - **Small VM instances**: Very small instance types (for example, `t2.micro`) can experience timeouts. Use a larger instance type such as `t2.small` or higher. +[1]: #linux-distributions-reference + {{% /tab %}} {{% tab "Docker" %}} @@ -44,19 +47,21 @@ Select your environment to see compatibility requirements and limitations: ### Compatibility - **Status**: GA -- **Supported operating systems**: See [Linux distributions reference](#linux-distributions-reference) +- **Supported operating systems**: See [Linux distributions reference][1] - **Supported architectures**: x86_64, arm64 ### Requirements - Datadog Agent with APM Instrumentation enabled -- Docker running on a [supported Linux distribution](#linux-distributions-reference) +- Docker running on a [supported Linux distribution][1] ### Limitations - **Rootless Docker mode**: When running Docker in rootless mode, update the socket path in `/etc/datadog-agent/inject/docker_config.yaml` so SSI can connect to Docker. The default path is `/run/user/$UID/docker.sock`, but your environment may differ. - **Custom `runc` shims**: If your environment uses custom `runc` shims (for example, for GPU workloads), update the `runtimes` entry in `/etc/datadog-agent/inject/docker_config.yaml` to include both your custom runtime and the Datadog runtime required for SSI. +[1]: #linux-distributions-reference + {{% /tab %}} {{% tab "Kubernetes" %}} @@ -66,13 +71,13 @@ Select your environment to see compatibility requirements and limitations: ### Compatibility - **Status**: GA -- **Supported node pools**: Linux nodes only (see [Linux distributions reference](#linux-distributions-reference)) +- **Supported node pools**: Linux nodes only (see [Linux distributions reference][2]) - **Supported architectures**: x86_64, arm64 ### Requirements - [Datadog Admission Controller][1] enabled -- Kubernetes nodes running a [supported Linux distribution](#linux-distributions-reference) +- Kubernetes nodes running a [supported Linux distribution][2] ### Limitations @@ -80,6 +85,7 @@ Select your environment to see compatibility requirements and limitations: - **Windows pods**: For Kubernetes clusters with Windows pods, use namespace inclusion/exclusion or specify an annotation in the application to exclude them from library injection. [1]: /containers/cluster_agent/admission_controller/ +[2]: #linux-distributions-reference {{% /tab %}} @@ -100,6 +106,30 @@ Select your environment to see compatibility requirements and limitations: - **IIS only**: Only .NET applications running in IIS are supported. +{{% /tab %}} + +{{% tab "Windows Host-wide (Preview)" %}} + +### Compatibility + +- **Status**: Preview; enrollment is required +- **Supported runtimes**: Java and .NET + +### Requirements + +- Datadog Agent v7.73 or higher +- Access to the Windows host-wide instrumentation Preview +- For .NET applications outside IIS, an instrumentation rule that identifies the process to instrument + +### Limitations + +- Host-wide instrumentation is available only to Preview participants. +- Language support is limited to Java and .NET. + +For enrollment and setup instructions, see [Windows Single Step Instrumentation][1]. + +[1]: /tracing/trace_collection/single-step-apm/windows/ + {{% /tab %}} {{< /tabs >}} @@ -181,7 +211,7 @@ To mitigate: [1]: /tracing/trace_collection/compatibility/python [2]: https://github.com/DataDog/dd-trace-py/releases -[3]: /tracing/trace_collection/automatic_instrumentation/single-step-apm/#instrument-sdks-across-applications +[3]: /tracing/trace_collection/single-step-apm/#instrument-sdks-across-applications [4]: /tracing/trace_collection/dd_libraries/python/ {{< /programming-lang >}} @@ -234,10 +264,6 @@ Supported Node.js versions are available by default only on: For other distributions, you may need to install Node.js separately. -### Limitations - -- **ESM modules**: Instrumentation of ESM (ECMAScript modules) is not supported. - [1]: /tracing/trace_collection/compatibility/nodejs {{< /programming-lang >}} diff --git a/content/en/tracing/trace_collection/single-step-apm/docker.md b/content/en/tracing/trace_collection/single-step-apm/docker.md index efcc360bf30..486ce540c99 100644 --- a/content/en/tracing/trace_collection/single-step-apm/docker.md +++ b/content/en/tracing/trace_collection/single-step-apm/docker.md @@ -1,5 +1,6 @@ --- title: Single Step APM Instrumentation on Docker +description: "Enable Datadog Single Step Instrumentation for applications in Docker containers, verify traces, configure SDK versions and tags, and remove instrumentation." code_lang: docker type: multi-code-lang code_lang_weight: 10 @@ -13,54 +14,93 @@ further_reading: ## Overview -In a Docker Linux container, use Single Step Instrumentation (SSI) for APM to install the Datadog Agent and [instrument][14] your applications in one step, with no additional configuration required. +For applications in Docker Linux containers, Single Step Instrumentation (SSI) configures the Docker host to [instrument][3] supported application containers without changing their images. ## Enable APM on your applications -
Before proceeding, confirm that your environment is compatible by reviewing the SSI compatibility guide.
+Before you begin: -To enable APM in a Docker Linux container: +- Confirm that your environment meets the [SSI compatibility requirements][15]. +- Run the commands in this section on the Docker host, not inside the Agent or application container. -1. In Datadog, go to the [Install the Datadog Agent on Docker][15] page. -1. In the {{< ui >}}Customize my agent install command{{< /ui >}} section, go to {{< ui >}}Additional configuration{{< /ui >}} > {{< ui >}}Application Observability{{< /ui >}}, and turn on {{< ui >}}APM Instrumentation{{< /ui >}}. - - {{< img src="tracing/trace_collection/docker-apm-instrumentation-toggle.png" alt="The 'Customize your agent install command' section of in-app instructions for installing the Datadog Agent on Docker" style="width:100%;" >}} +{{< tabs >}} +{{% tab "Installation command" %}} -1. Copy and run the Agent installation command in your Docker container. If the Agent is already running, redeploy the Agent container using the new command. -1. Restart your applications. +1. Configure the Docker host to inject Datadog SDKs into new containers: + + ```shell + DD_APM_INSTRUMENTATION_ENABLED=docker DD_NO_AGENT_INSTALL=true \ + bash -c "$(curl -L https://install.datadoghq.com/scripts/install_script_agent7.sh)" + ``` + + This installs the Datadog container injector and configures the Docker runtime. It does not install or replace the Agent container. + +1. Start the Agent with APM enabled and share the injector's socket directory. Replace `` with your [Datadog API key][1] and `` with your [Datadog site][2]: + + ```shell + docker run -d --name dd-agent \ + -e DD_API_KEY= \ + -e DD_SITE= \ + -e DD_APM_ENABLED=true \ + -e DD_APM_NON_LOCAL_TRAFFIC=true \ + -e DD_APM_RECEIVER_SOCKET=/opt/datadog/apm/inject/run/apm.socket \ + -v /opt/datadog/apm:/opt/datadog/apm \ + -v /var/run/docker.sock:/var/run/docker.sock:ro \ + registry.datadoghq.com/agent:7 + ``` + + If the Agent container already exists, recreate it with equivalent APM socket and volume settings. Keep any other settings required by your environment. + +{{% /tab %}} +{{% tab "In-app instructions" %}} + +1. In Datadog, go to the [Install the Datadog Agent on Docker][4] page. +1. In the {{< ui >}}Customize my Agent install command{{< /ui >}} section, go to {{< ui >}}Additional configuration{{< /ui >}} > {{< ui >}}Application Observability{{< /ui >}}, and turn on {{< ui >}}APM Instrumentation{{< /ui >}}. + + {{< img src="tracing/trace_collection/docker-apm-instrumentation-toggle.png" alt="The 'Customize your Agent install command' section of in-app instructions for installing the Datadog Agent on Docker" style="width:100%;" >}} + +1. Copy and run the generated commands on the Docker host. If the Agent is already running, recreate the Agent container with the generated configuration. + +{{% /tab %}} +{{< /tabs >}} + +(Optional) By default, SSI installs the latest SDK major versions. To pin specific versions, see [Set SDK tracer versions][17].
SSI adds a small amount of startup time to instrumented applications. If this overhead is not acceptable for your use case, contact Datadog Support.
-## Set SDK tracer versions +## Verify your first trace -By default, Single Step Instrumentation installs the latest major versions of Datadog SDKs. Minor version updates are applied automatically when they become available. +1. Recreate or restart your application containers so SSI can inject the Datadog SDK when each process starts. +1. Generate traffic to your application. +1. In Datadog, go to [**APM** > **Traces**][5] to confirm that your service is reporting traces. -You may want to customize SDK versions based on your application's language version or specific environment requirements. You can control the major and minor versions used by customizing library versions during setup. +If traces don't appear: -To customize tracer versions: +- Confirm that the Agent and application containers share `/opt/datadog/apm` and that the Agent APM socket exists in that directory. +- Set `DD_APM_INSTRUMENTATION_DEBUG=true` on the application container, recreate it, then inspect `docker logs ` for injection messages. +- Inspect the application process environment inside the container for the language-specific settings described in the [injector guide][16]. -1. In Datadog, go to the [Install the Datadog Agent on Docker][15] page. -1. After you turn on {{< ui >}}APM Instrumentation{{< /ui >}}, click {{< ui >}}Customize library versions{{< /ui >}}. +For more help, see the [SSI troubleshooting guide][6]. - {{< img src="tracing/trace_collection/apm-instrumentation-version-pinning.png" alt="The 'Customize library versions' drop-down in the instructions for installing the Datadog Agent on Docker" style="width:100%;" >}} +## Set SDK tracer versions -1. Find your language(s) and use the dropdown to either: - - Pin an exact tracer version, or - - Select the major version you want to use. -1. Copy and run the updated installation command. +By default, Single Step Instrumentation installs the latest major versions of Datadog SDKs, and applies minor version updates automatically when they become available. -Available versions are listed in source repositories for each language: +To pin specific major or exact versions, set the `DD_APM_INSTRUMENTATION_LIBRARIES` variable in the install command: -- [Java][8] (`java`) -- [Node.js][9] (`js`) -- [Python][10] (`python`) -- [.NET][11] (`dotnet`) -- [Ruby][12] (`ruby`) -- [PHP][13] (`php`) +```shell +DD_APM_INSTRUMENTATION_ENABLED=docker DD_NO_AGENT_INSTALL=true \ + DD_APM_INSTRUMENTATION_LIBRARIES="java:1,python:4,js:5,dotnet:3,ruby:2,php:1" \ + bash -c "$(curl -L https://install.datadoghq.com/scripts/install_script_agent7.sh)" +``` + +Alternatively, use the in-app installer: after you turn on {{< ui >}}APM Instrumentation{{< /ui >}} on the [Install the Datadog Agent on Docker][4] page, click {{< ui >}}Customize library versions{{< /ui >}} and select an exact or major version for each language. + +Available versions are listed in source repositories for each language: [Java][8] (`java`), [Node.js][9] (`js`), [Python][10] (`python`), [.NET][11] (`dotnet`), [Ruby][12] (`ruby`), [PHP][13] (`php`). ## Configure Unified Service Tags -Unified Service Tags (USTs) apply consistent tags across traces, metrics, and logs, making it easier to navigate and correlate your observability data. Learn how to [set USTs for Docker services][16]. +Unified Service Tags (USTs) apply consistent tags across traces, metrics, and logs, making it easier to navigate and correlate your observability data. Learn how to [set USTs for Docker services][7]. ## Enable SDK-dependent products and features @@ -68,11 +108,11 @@ After SSI loads the Datadog SDK into your applications and enables distributed t {{< ssi-products >}} -To enable products, [set environment variables][3] in your application configuration. +To enable products, [set environment variables][14] in your application configuration. ## Remove Single Step APM instrumentation from your Agent -If you don't want to collect trace data for a particular service, host, VM, or container, complete the following steps: +If you don't want to collect trace data for a particular service, host, VM, or container, complete the following steps. ### Remove instrumentation for specific services @@ -88,7 +128,7 @@ To remove APM instrumentation and stop sending traces from a specific service: To stop producing traces, uninstall APM and restart the infrastructure: -1. Run: +1. On the Docker host, run: ```shell dd-container-install --uninstall ``` @@ -100,25 +140,26 @@ To stop producing traces, uninstall APM and restart the infrastructure: ## Troubleshooting -If you encounter problems enabling APM with SSI, see the [SSI troubleshooting guide][17]. +If you encounter problems enabling APM with SSI, see the [SSI troubleshooting guide][6]. ## Further reading {{< partial name="whats-next/whats-next.html" >}} [1]: https://app.datadoghq.com/organization-settings/api-keys -[2]: /tracing/trace_collection/library_config/ -[3]: /tracing/trace_collection/library_config/ +[2]: /getting_started/site/ +[3]: /tracing/glossary/#instrumentation +[4]: https://app.datadoghq.com/fleet/install-agent/latest?platform=docker +[5]: https://app.datadoghq.com/apm/traces +[6]: /tracing/trace_collection/single-step-apm/troubleshooting/ +[7]: /getting_started/tagging/unified_service_tagging/?tab=docker#containerized-environment [8]: https://github.com/DataDog/dd-trace-java/releases [9]: https://github.com/DataDog/dd-trace-js/releases [10]: https://github.com/DataDog/dd-trace-py/releases [11]: https://github.com/DataDog/dd-trace-dotnet/releases [12]: https://github.com/DataDog/dd-trace-rb/releases [13]: https://github.com/DataDog/dd-trace-php/releases -[14]: /tracing/glossary/#instrumentation -[15]: https://app.datadoghq.com/fleet/install-agent/latest?platform=docker -[16]: /getting_started/tagging/unified_service_tagging/?tab=docker#containerized-environment -[17]: /tracing/trace_collection/automatic_instrumentation/single-step-apm/troubleshooting - - - +[14]: /tracing/trace_collection/library_config/ +[15]: /tracing/trace_collection/single-step-apm/compatibility/ +[16]: /tracing/guide/injectors/#per-runtime-instrumentation +[17]: #set-sdk-tracer-versions diff --git a/content/en/tracing/trace_collection/single-step-apm/kubernetes.md b/content/en/tracing/trace_collection/single-step-apm/kubernetes.md index 0b9c791c936..518610c1582 100644 --- a/content/en/tracing/trace_collection/single-step-apm/kubernetes.md +++ b/content/en/tracing/trace_collection/single-step-apm/kubernetes.md @@ -1,5 +1,6 @@ --- title: Single Step APM Instrumentation on Kubernetes +description: "Enable Single Step Instrumentation (SSI) on Kubernetes to automatically instrument applications with Datadog APM. Covers prerequisites, setup with Helm or the Datadog Operator, verification, Unified Service Tags, workload targeting, and troubleshooting." code_lang: kubernetes type: multi-code-lang code_lang_weight: 20 @@ -12,851 +13,652 @@ further_reading: - link: /tracing/guide/init_resource_calc/ tag: Documentation text: Learn about init container resource usage - - link: /tracing/guide/local_sdk_injection + - link: /tracing/trace_collection/single-step-apm/troubleshooting tag: Documentation - text: Instrument your applications using local SDK injection - - link: "https://learn.datadoghq.com/courses/configuring-ssi-k8s" - tag: "Learning Center" - text: "Configuring Single Step Instrumentation on Kubernetes" - + text: Troubleshoot Single Step APM + - link: /tracing/guide/injectors/ + tag: Documentation + text: Understanding injector behavior with Single Step Instrumentation --- ## Overview -In a Kubernetes environment, use Single Step Instrumentation (SSI) for APM to install the Datadog Agent and [instrument][3] your applications with the Datadog SDKs in one step. - -## Requirements - -- Kubernetes v1.20+. -- [`Helm`][1] for deploying the Datadog Operator. -- [`Kubectl` CLI][2] for installing the Datadog Agent. -- Confirmed environment compatibility per the [Single Step Instrumentation compatibility guide][36]. - - -## Enable APM on your applications - -
Single Step Instrumentation does not instrument applications in the namespace where the Datadog Agent is installed. Install the Agent in a separate namespace where you do not run your applications.
- -Follow these steps to enable Single Step Instrumentation across your entire cluster. This automatically sends traces from all applications written in supported languages. +Single Step Instrumentation (SSI) installs the Datadog Agent and instruments your Kubernetes applications in one step. SSI loads the Datadog SDK into your application processes at runtime, with no code changes or image rebuilds required. After you enable SSI, all supported applications in your cluster automatically begin sending traces to Datadog. -**Note:** To instrument only specific namespaces or pods, see workload targeting in [Advanced options](#advanced-options). +## Prerequisites -1. In Datadog, go to the [Install the Datadog Agent on Kubernetes][11] page. -1. Follow the on-screen instructions to choose your installation method, select an API key, and set up the Operator or Helm repository. -1. In the {{< ui >}}Configure `datadog-agent.yaml`{{< /ui >}} section, go to {{< ui >}}Additional configuration{{< /ui >}} > {{< ui >}}Application Observability{{< /ui >}}, and turn on {{< ui >}}APM Instrumentation{{< /ui >}}. +- Kubernetes v1.20+ +- [Helm][1] for installing the Datadog Agent or Datadog Operator +- [kubectl][2] for applying manifests and restarting workloads +- A supported language runtime per the [SSI compatibility guide][3] - {{< img src="tracing/trace_collection/k8s-apm-instrumentation-toggle.jpg" alt="The configuration block for installing the Datadog Agent on Kubernetes through the Datadog app" style="width:100%;" >}} +### Check for existing tracer dependencies -1. Deploy the Agent using the generated configuration file. -1. Restart your applications. +Before you enable SSI, check whether your application already loads a tracing SDK. Search dependency manifests, lock files, Dockerfiles, and startup scripts: -
SSI adds a small amount of startup time to instrumented applications. If this overhead is not acceptable for your use case, contact Datadog Support.
- -## Configure Unified Service Tags - -Unified Service Tags (USTs) apply consistent tags across traces, metrics, and logs, making it easier to navigate and correlate your observability data. You can configure USTs through automatic label extraction (recommended), through explicit configuration with `ddTraceConfigs`, or in deployment manifests. - -
-If you are using Remote Configuration, automatic label extraction is not compatible. You must configure USTs explicitly using ddTraceConfigs. -
+```shell +grep -RniE "ddtrace|dd-trace|dd-java-agent|javaagent|opentelemetry" \ + --include='requirements*.txt' --include='pyproject.toml' --include='poetry.lock' \ + --include='package*.json' --include='yarn.lock' --include='pnpm-lock.yaml' \ + --include='Gemfile*' --include='pom.xml' --include='build.gradle*' \ + --include='*.csproj' --include='Dockerfile*' . +``` -### (Recommended) Configure USTs through automatic label extraction +For Java, also check Dockerfiles and startup scripts for `-javaagent` flags, and check the `JAVA_TOOL_OPTIONS` environment variable. -With SSI, you can automatically extract UST values from pod labels and metadata without modifying individual deployments. To do this, configure `kubernetesResourcesLabelsAsTags` to map your existing Kubernetes labels to Datadog service tags. +Review each match before making changes. Remove or disable only conflicting tracer initialization and dependencies, then rebuild the application image. Do not remove OpenTelemetry API, metrics, or logging dependencies solely because they match the search. -**Note:** This method is not compatible with Remote Configuration. If you're using Remote Configuration, see [Configure USTs explicitly with ddTraceConfigs](#configure-usts-explicitly-with-ddtraceconfigs). +### Compatibility notes -#### Prerequisites +- **Ruby**: SSI requires glibc and is not compatible with Alpine or other musl-based images. +- **All other languages**: Alpine and musl-based images are supported on Kubernetes. SSI injects through the `LD_PRELOAD` environment variable, not `/etc/ld.so.preload`. -| Component | Minimum version | -|-----------|------------------| -| `datadog-agent` | 7.69 | -| `datadog-operator` | 1.16.0 | -| `datadog-helm-chart` | 3.120.0 | +## Enable SSI -#### Configuration +
SSI does not instrument applications in the namespace where the Datadog Agent is installed. Install the Agent in a separate namespace.
-Replace `app.kubernetes.io/name` in the following example with any label that contains your service name (for example, `service.kubernetes.io/name` or `component`). You can configure multiple labels this way. +These steps enable SSI across your entire cluster. To instrument specific namespaces or pods, see [Target specific workloads][12]. -```yaml -datadog: - # Automatically extract service names from Kubernetes labels - kubernetesResourcesLabelsAsTags: - pods: - app.kubernetes.io/name: service # Modern Kubernetes label - deployments.apps: - app.kubernetes.io/name: service - replicasets.apps: - app.kubernetes.io/name: service +{{< tabs >}} +{{% tab "Helm" %}} - # Set environment globally for the entire cluster - tags: - - "env:production" +If the Datadog Agent is not installed, set `DD_API_KEY` to your [Datadog API key][10]. Then add the Datadog Helm repository and create a Kubernetes Secret: - apm: - instrumentation: - enabled: true +```shell +helm repo add datadog https://helm.datadoghq.com +helm repo update +kubectl create namespace datadog --dry-run=client -o yaml | kubectl apply -f - +kubectl create secret generic datadog-secret --from-literal api-key=$DD_API_KEY -n datadog --dry-run=client -o yaml | kubectl apply -f - ``` -With this configuration, Datadog automatically sets the `service` tag using the value of the `app.kubernetes.io/name` label for any instrumented workload that includes this label. +If you install the Agent in a different namespace, replace `datadog` with your Agent namespace. -### Configure USTs explicitly with ddTraceConfigs +If the Agent is already installed, add `datadog.apm.instrumentation.enabled: true` to your existing `datadog-values.yaml` and skip to the `helm upgrade` step. -In most cases, automatic configuration is sufficient. However, if you need granular control over settings for specific workloads, use `ddTraceConfigs` to explicitly map labels to service configurations: +Create a `datadog-values.yaml`: ```yaml datadog: - kubernetesResourcesLabelsAsTags: - pods: - app.kubernetes.io/name: service - deployments.apps: - app.kubernetes.io/name: service - - # Set environment globally for the entire cluster - tags: - - "env:production" - + apiKeyExistingSecret: datadog-secret + clusterName: + site: apm: instrumentation: enabled: true - targets: - - name: frontend-services - podSelector: - matchLabels: - tier: frontend - ddTraceConfigs: - - name: DD_SERVICE # Explicitly override service name - valueFrom: - fieldRef: - fieldPath: metadata.labels['app.kubernetes.io/name'] - # DD_ENV inherited from cluster-level tags above - # DD_VERSION automatically extracted from image tags ``` +Replace `` with your Kubernetes cluster name and `` with your [Datadog site][11]. -### Configure USTs in deployment manifests - -If your setup doesn't use labels suitable for UST extraction, you can set USTs directly in your deployment manifests using environment variables. This approach requires modifying each deployment individually, but offers precise control. +Deploy or update the Agent: -For complete instructions, see [setting USTs for Kubernetes services][5]. +```shell +helm upgrade --install datadog-agent -f datadog-values.yaml datadog/datadog -n datadog +``` -## Enable SDK-dependent products and features +SSI installs the latest SDK for each supported language by default. To pin SDK versions, see [Pin SDK versions][13]. -After SSI loads the Datadog SDK into your applications and enables distributed tracing, you can configure additional products that rely on the SDK: +Restart your application pods: -{{< ssi-products >}} +
Restarting pods can cause a brief service interruption. Coordinate with the application owner before you proceed.
-Use one of the following setup methods: +```shell +kubectl rollout restart deployment/ -n +``` -- **[Configure with workload targeting (recommended)](#target-specific-workloads)**: +For a StatefulSet or DaemonSet, replace `deployment/` with the controller type and name. - By default, Single Step Instrumentation instruments all services in all namespaces. Use workload targeting to limit instrumentation to specific namespaces, pods, or workloads, and apply custom configurations. +{{% /tab %}} +{{% tab "Datadog Operator" %}} -- **[Set environment variables][7]**: +If the Datadog Operator and Agent are not installed, set `DD_API_KEY` to your [Datadog API key][10]. Then install the Operator and create a Kubernetes Secret: - Enable products by setting environment variables directly in your application configuration. +```shell +helm repo add datadog https://helm.datadoghq.com +helm repo update +helm upgrade --install datadog-operator datadog/datadog-operator --namespace datadog --create-namespace +kubectl create secret generic datadog-secret --from-literal api-key=$DD_API_KEY -n datadog --dry-run=client -o yaml | kubectl apply -f - +``` -## Advanced options +If you install the Agent in a different namespace, replace `datadog` with your Agent namespace. -Use the following advanced options to customize how Single Step Instrumentation behaves in your environment. These settings are optional and typically only needed in specialized setups. +If the Agent is already installed with the Datadog Operator, add `features.apm.instrumentation.enabled: true` to your existing `DatadogAgent` manifest and skip to the `kubectl apply` step. -### Configure injection modes +Create a `datadog-agent.yaml`: -SSI supports multiple injection modes, which control how the injector and APM library files are delivered to your application containers. You typically do not need to configure this setting manually. Consider adjusting it if you notice significant pod startup delays or higher-than-expected resource usage (CPU, memory) during pod initialization. For more on how the injector works, see [Injector Behavior with Single Step Instrumentation][41]. +```yaml +apiVersion: datadoghq.com/v2alpha1 +kind: DatadogAgent +metadata: + name: datadog + namespace: datadog +spec: + global: + clusterName: + site: + credentials: + apiSecret: + secretName: datadog-secret + keyName: api-key + features: + apm: + instrumentation: + enabled: true +``` +Replace `` with your Kubernetes cluster name and `` with your [Datadog site][11]. -| Mode | Description | Requirements | -|------|-------------|--------------| -| `init_container` | Uses init containers to copy injector and APM library files into application containers. | Agent deployed with Helm Chart or Datadog Operator | -| `csi` | **In Preview.** Mounts injector and APM library files using the [Datadog CSI driver][37]. Reduces pod startup time compared to init container mode. | Agent 7.76.0+, CSI driver 1.2.0+, Helm Chart 3.178.1+ or Datadog Operator 1.25.0+ | +Apply the manifest: -Before using `csi` mode, install and activate the Datadog CSI driver. If you are deploying with Helm, also set `datadog.csi.enabled: true` in your `datadog-values.yaml`. See the [CSI driver documentation][37] for installation steps and environment-specific requirements such as GKE Autopilot. +```shell +kubectl apply -f datadog-agent.yaml +``` -#### Configure injection mode globally +SSI installs the latest SDK for each supported language by default. To pin SDK versions, see [Pin SDK versions][13]. -{{< tabs >}} -{{% tab "Helm" %}} +Restart your application pods: -To set the injection mode cluster-wide, add `injectionMode` to your `datadog-values.yaml`: +
Restarting pods can cause a brief service interruption. Coordinate with the application owner before you proceed.
-```yaml -datadog: - apm: - instrumentation: - injectionMode: +```shell +kubectl rollout restart deployment/ -n ``` -Supported values: `init_container`, `csi`. +For a StatefulSet or DaemonSet, replace `deployment/` with the controller type and name. {{% /tab %}} -{{% tab "Datadog Operator" %}} +{{% tab "In-app wizard" %}} -To set the injection mode cluster-wide, add `injectionMode` to your `datadog-agent.yaml`: +The Datadog in-app wizard generates a configuration file with SSI enabled: -```yaml -features: - apm: - instrumentation: - injectionMode: -``` +1. Go to the [Install the Datadog Agent on Kubernetes][14] page. +1. Choose your installation method, select an API key, and set up the Operator or Helm repository. +1. Under **Configure `datadog-agent.yaml`**, go to **Additional configuration** > **Application Observability** and turn on **APM Instrumentation**. + + {{< img src="tracing/trace_collection/k8s-apm-instrumentation-toggle.jpg" alt="APM Instrumentation toggle in the Kubernetes Agent installation wizard" style="width:100%;" >}} -Supported values: `init_container`, `csi`. +1. Deploy the Agent with the generated configuration file. +1. Restart your application pods: -If you are using Datadog Operator earlier than 1.25.0, use the [pod annotation](#configure-injection-mode-per-pod) to override the injection mode for specific pods. +
Restarting pods can cause a brief service interruption. Coordinate with the application owner before you proceed.
+ + ```shell + kubectl rollout restart deployment/ -n + ``` {{% /tab %}} {{< /tabs >}} -#### Configure injection mode per pod +
SSI adds a small amount of startup time to instrumented applications. If this overhead is not acceptable, contact Datadog Support.
-To override the injection mode for a specific pod, add the following annotation to the pod spec: +
If your cluster uses init-container injection and enforces PodSecurity restricted policies, the Datadog init container may be blocked with allowPrivilegeEscalation is false or violates PodSecurity "restricted:latest". See the SSI troubleshooting guide for the required security context configuration.
-```yaml -metadata: - annotations: - admission.datadoghq.com/apm-inject.injection-mode: "" -``` - -Supported values: `init_container`, `csi`. - -### Target specific workloads +## Verify SSI is working -By default, SSI instruments all services in all namespaces in your cluster. Depending on your Agent version, use one of the following configuration methods to refine which services are instrumented and how. +After you restart your application pods, wait two to three minutes for traces to arrive, then confirm SSI is working: -{{< tabs >}} +1. **Check that the pod was mutated.** The expected evidence depends on the configured [injection mode][15]: -{{% tab "Agent v7.64+ (Recommended)" %}} + - For `init_container`, the output of this command includes `datadog-lib--init`: -Create targeting blocks with the `targets` label to specify which workloads to instrument and what configurations to apply. + ```shell + kubectl get pod -n -o jsonpath='{.spec.initContainers[*].name}' + ``` -Each target block has the following keys: + - For `csi` or `image_volume`, an SDK init container is not expected. Describe the pod and confirm that it has Datadog-injected volumes and mounts, and that its events contain no mount errors: -| Key | Description | -|------------------|-------------| -| `name` | The name of the target block. This has no effect on monitoring state and is used only as metadata. | -| `namespaceSelector` | The namespace(s) to instrument. Specify using one or more of:
- `matchNames`: A list of one or more namespace name(s).
- `matchLabels`: A list of one or more label(s) defined in `{key,value}` pairs.
- `matchExpressions`: A list of namespace selector requirements.

Namespaces must meet all criteria to match. For more details, see the [Kubernetes selector documentation][10].| -| `podSelector` | The pod(s) to instrument. Specify using one or more of:
- `matchLabels`: A list of one or more label(s) defined in `{key,value}` pairs.
- `matchExpressions`: A list of pod selector requirements.

Pods must meet all criteria to match. For more details, see the [Kubernetes selector documentation][10]. | -| `ddTraceVersions` | The [Datadog APM SDK][9] version to use for each language. | -| `ddTraceConfigs` | APM SDK configs that allow setting [Unified Service Tags][8], enabling [SDK-dependent products](#enable-sdk-dependent-products-and-features) beyond tracing, and customizing other [APM settings][14]. | + ```shell + kubectl describe pod -n + ``` -The file you need to configure depends on how you enabled Single Step Instrumentation: -- If you enabled SSI with Datadog Operator, edit `datadog-agent.yaml`. -- If you enabled SSI with Helm, edit `datadog-values.yaml`. + When `injectionMode` is `auto`, inspect the mutated pod to determine which delivery method the Cluster Agent selected. -**Note**: Targets are evaluated in order; the first match takes precedence. +2. **Generate traffic.** If your application does not receive traffic automatically, port-forward and send a few requests: + ```shell + kubectl port-forward deployment/ 8080: -n + curl http://localhost:8080/ + ``` -#### Example configurations +3. **Find your service.** Go to [**APM** > **Catalog**][6] and confirm your service appears. -Review the following examples demonstrating how to select specific services: +4. **Confirm traces.** Go to [**APM** > **Traces**][7] and search for traces from your service. -{{< collapse-content title="Example 1: Enable all namespaces except one" level="h4" >}} +If traces do not appear after five minutes, see the [SSI troubleshooting guide][8]. -This configuration: -- enables APM for all namespaces except the `jenkins` namespace. - - **Note**: use `enabledNamespaces` to disable for all namespaces except those listed. -- instructs Datadog to instrument the Java applications with the default Java SDK and Python applications with `v.3.1.0` of the Python SDK. +## Configure Unified Service Tags -{{< highlight yaml "hl_lines=4-10" >}} - apm: - instrumentation: - enabled: true - disabledNamespaces: - - "jenkins" - targets: - - name: "all-remaining-services" - ddTraceVersions: - java: "default" - python: "3.1.0" -{{< /highlight >}} +[Unified Service Tags][5] (USTs) apply consistent `service`, `env`, and `version` tags across traces, metrics, and logs. SSI can automatically extract UST values from your existing Kubernetes labels. -{{< /collapse-content >}} +
+Automatic label extraction is not compatible with Remote Configuration. If you use Remote Configuration, configure USTs with ddTraceConfigs instead. +
-{{< collapse-content title="Example 2: Instrument a subset of namespaces, matching on names and labels" level="h4" >}} +### Automatic label extraction (recommended) -This configuration creates two targets blocks: +Map existing Kubernetes labels to the `service`, `env`, and `version` tags with `kubernetesResourcesLabelsAsTags`. Replace the example label keys with the labels used by your workloads. -- The first block (named `login-service_namespace`): - - enables APM for services in the namespace `login-service`. - - instructs Datadog to instrument services in this namespace with the default version of the Java SDK. - - sets environment variable `DD_PROFILING_ENABLED` for this target group -- The second block (named `billing-service_apps`) - - enables APM for services in the namespace(s) with label `app:billing-service`. - - instructs Datadog to instrument this set of services with `v3.1.0` of the Python SDK. +{{< tabs >}} +{{% tab "Helm" %}} -{{< highlight yaml "hl_lines=4-28" >}} +```yaml +datadog: + kubernetesResourcesLabelsAsTags: + pods: + app.kubernetes.io/name: service + app.kubernetes.io/environment: env + app.kubernetes.io/version: version + deployments.apps: + app.kubernetes.io/name: service + app.kubernetes.io/environment: env + app.kubernetes.io/version: version + replicasets.apps: + app.kubernetes.io/name: service + app.kubernetes.io/environment: env + app.kubernetes.io/version: version apm: instrumentation: enabled: true - targets: - - name: "login-service_namespace" - namespaceSelector: - matchNames: - - "login-service" - ddTraceVersions: - java: "default" - ddTraceConfigs: - - name: "DD_PROFILING_ENABLED" ## profiling is enabled for all services in this namespace - value: "auto" - - name: "billing-service_apps" - namespaceSelector: - matchLabels: - app: "billing-service" - ddTraceVersions: - python: "3.1.0" -{{< /highlight >}} - -{{< /collapse-content >}} - -{{< collapse-content title="Example 3: Instrument different workloads with different tracers" level="h4" >}} - -This configuration does the following: -- enables APM for pods with the following labels: - - `app:db-user`, which marks pods running the `db-user` application. - - `webserver:routing`, which marks pods running the `request-router` application. -- instructs Datadog to use the default versions of the Datadog Tracer SDKs. -- sets Datadog environment variables to apply to each target group and configure the SDKs. - -{{< highlight yaml "hl_lines=4-28" >}} - apm: - instrumentation: - enabled: true - targets: - - name: "db-user" - podSelector: - matchLabels: - app: "db-user" - ddTraceVersions: - java: "default" - ddTraceConfigs: ## trace configs set for services in matching pods - - name: "DD_DATA_STREAMS_ENABLED" - value: "true" - - name: "user-request-router" - podSelector: - matchLabels: - webserver: "user" - ddTraceVersions: - php: "default" -{{< /highlight >}} - -{{< /collapse-content >}} - -{{< collapse-content title="Example 4: Instrument a pod within a namespace" level="h4" >}} - -This configuration: -- enables APM for pods labeled `app:password-resolver` inside the `login-service` namespace. -- instructs Datadog to use the default version of the Datadog Java Tracer SDK. -- sets Datadog environment variables to apply to this target. - -{{< highlight yaml "hl_lines=4-28" >}} - apm: - instrumentation: - enabled: true - targets: - - name: "login-service-namespace" - namespaceSelector: - matchNames: - - "login-service" - podSelector: - matchLabels: - app: "password-resolver" - ddTraceVersions: - java: "default" - ddTraceConfigs: - - name: "DD_PROFILING_ENABLED" - value: "auto" -{{< /highlight >}} - -{{< /collapse-content >}} - -{{< collapse-content title="Example 5: Instrument a subset of pods using matchExpressions" level="h4" >}} - -This configuration enables APM for all pods except those that have either of the labels `app=app1` or `app=app2`. - -{{< highlight yaml "hl_lines=4-28" >}} - apm: - instrumentation: - enabled: true - targets: - - name: "default-target" - podSelector: - matchExpressions: - - key: app - operator: NotIn - values: - - app1 - - app2 -{{< /highlight >}} - -{{< /collapse-content >}} - -{{< collapse-content title="Example 6: Enable additional products with ddTraceConfigs" level="h4" >}} - -This configuration enables [App and API Protection (AAP)][12] and [Continuous Profiler][11] for services in the `web-apps` namespace, using `ddTraceConfigs` to set the required environment variables: - -{{< highlight yaml "hl_lines=4-20" >}} - apm: - instrumentation: - enabled: true - targets: - - name: "web-apps-with-security" - namespaceSelector: - matchNames: - - "web-apps" - ddTraceVersions: - java: "default" - python: "default" - ddTraceConfigs: - - name: "DD_APPSEC_ENABLED" - value: "true" - - name: "DD_PROFILING_ENABLED" - value: "auto" -{{< /highlight >}} - -For a full list of products you can enable through SSI, see [Enable SDK-dependent products and features](#enable-sdk-dependent-products-and-features). - -{{< /collapse-content >}} - -[8]: /getting_started/tagging/unified_service_tagging/?tab=kubernetes -[9]: /tracing/trace_collection/automatic_instrumentation/single-step-apm/compatibility/#tracer-libraries -[10]: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#resources-that-support-set-based-requirements -[11]: /profiler/ -[12]: /security/application_security/ -[14]: /tracing/trace_collection/library_config/ +``` {{% /tab %}} +{{% tab "Datadog Operator" %}} -{{% tab "Agent <=v7.63 (Legacy)" %}} +```yaml +spec: + global: + kubernetesResourcesLabelsAsTags: + pods: + app.kubernetes.io/name: service + app.kubernetes.io/environment: env + app.kubernetes.io/version: version + deployments.apps: + app.kubernetes.io/name: service + app.kubernetes.io/environment: env + app.kubernetes.io/version: version + replicasets.apps: + app.kubernetes.io/name: service + app.kubernetes.io/environment: env + app.kubernetes.io/version: version + features: + apm: + instrumentation: + enabled: true +``` -#### Enable or disable instrumentation for namespaces +{{% /tab %}} +{{< /tabs >}} -You can choose to enable or disable instrumentation for applications in specific namespaces. You can only set enabledNamespaces or disabledNamespaces, not both. +Automatic label extraction requires Agent v7.69 or later and either Helm chart v3.120.0 or later or Datadog Operator v1.16.0 or later. -The file you need to configure depends on if you enabled Single Step Instrumentation with Datadog Operator or Helm: +{{% collapse-content title="Configure Unified Service Tags with ddTraceConfigs" level="h3" expanded=false %}} -{{< collapse-content title="Datadog Operator" level="h5" >}} +For granular control over specific workloads, use `ddTraceConfigs` to map labels to service configurations: -To enable instrumentation for specific namespaces, add `enabledNamespaces` configuration to `datadog-agent.yaml`: +{{< tabs >}} +{{% tab "Helm" %}} -{{< highlight yaml "hl_lines=5-7" >}} - features: - apm: - instrumentation: - enabled: true - enabledNamespaces: # Add namespaces to instrument - - default - - applications -{{< /highlight >}} +```yaml +datadog: + apm: + instrumentation: + enabled: true + targets: + - name: frontend-services + podSelector: + matchLabels: + tier: frontend + ddTraceConfigs: + - name: DD_SERVICE + valueFrom: + fieldRef: + fieldPath: metadata.labels['app.kubernetes.io/name'] + - name: DD_ENV + valueFrom: + fieldRef: + fieldPath: metadata.labels['app.kubernetes.io/environment'] + - name: DD_VERSION + valueFrom: + fieldRef: + fieldPath: metadata.labels['app.kubernetes.io/version'] +``` -To disable instrumentation for specific namespaces, add `disabledNamespaces` configuration to `datadog-agent.yaml`: +{{% /tab %}} +{{% tab "Datadog Operator" %}} -{{< highlight yaml "hl_lines=5-7" >}} - features: - apm: - instrumentation: - enabled: true - disabledNamespaces: # Add namespaces to not instrument - - default - - applications -{{< /highlight >}} +```yaml +spec: + features: + apm: + instrumentation: + enabled: true + targets: + - name: frontend-services + podSelector: + matchLabels: + tier: frontend + ddTraceConfigs: + - name: DD_SERVICE + valueFrom: + fieldRef: + fieldPath: metadata.labels['app.kubernetes.io/name'] + - name: DD_ENV + valueFrom: + fieldRef: + fieldPath: metadata.labels['app.kubernetes.io/environment'] + - name: DD_VERSION + valueFrom: + fieldRef: + fieldPath: metadata.labels['app.kubernetes.io/version'] +``` -{{< /collapse-content >}} +{{% /tab %}} +{{< /tabs >}} -{{< collapse-content title="Helm" level="h5" >}} +{{% /collapse-content %}} -To enable instrumentation for specific namespaces, add `enabledNamespaces` configuration to `datadog-values.yaml`: +{{% collapse-content title="Configure Unified Service Tags in deployment manifests" level="h3" expanded=false %}} -{{< highlight yaml "hl_lines=5-7" >}} - datadog: - apm: - instrumentation: - enabled: true - enabledNamespaces: # Add namespaces to instrument - - namespace_1 - - namespace_2 -{{< /highlight >}} +If your labels are not suitable for automatic extraction, set USTs directly in your deployment manifests with environment variables. This requires modifying each deployment individually. -To disable instrumentation for specific namespaces, add `disabledNamespaces` configuration to `datadog-values.yaml`: +For complete instructions, see [setting USTs for Kubernetes services][5]. -{{< highlight yaml "hl_lines=5-7" >}} - datadog: - apm: - instrumentation: - enabled: true - disabledNamespaces: # Add namespaces to not instrument - - namespace_1 - - namespace_2 -{{< /highlight >}} +{{% /collapse-content %}} -{{< /collapse-content >}} +## Enable additional products -#### Specify SDK versions +After SSI enables distributed tracing, you can activate additional SDK-dependent products: -
Starting with Datadog Cluster Agent v7.52.0+, you can automatically instrument a subset of your applications, based on the SDKs you specify.
+{{< ssi-products >}} -Specify Datadog SDKs and their versions to automatically instrument applications written in those languages. You can configure this in two ways, which are applied in the following order of precedence: +To enable products: -1. [Specify at the service level](#specify-at-the-service-level), or -2. [Specify at the cluster level](#specify-at-the-cluster-level). +- **With workload targeting (recommended):** Add `ddTraceConfigs` entries to your target blocks. See [Target specific workloads][12]. +- **With environment variables:** Set variables directly in your application configuration. See [Library Configuration][9]. -**Default**: If you don't specify any library versions, applications written in supported languages are automatically instrumented using the latest SDK versions. +## Next steps -##### Specify at the service level +After traces are flowing: -To automatically instrument applications in specific pods, add the appropriate language annotation and library version for your application in your pod spec: +{{< whatsnext desc=" " >}} + {{< nextlink href="/tracing/trace_explorer/" >}}Trace Explorer: search and analyze your traces{{< /nextlink >}} + {{< nextlink href="/tracing/services/service_page/" >}}Service Page: monitor service health and performance{{< /nextlink >}} + {{< nextlink href="/tracing/trace_collection/custom_instrumentation/" >}}Custom instrumentation: add application-specific spans{{< /nextlink >}} + {{< nextlink href="/tracing/trace_collection/dynamic_instrumentation/" >}}Dynamic Instrumentation: add custom spans without redeploying{{< /nextlink >}} +{{< /whatsnext >}} -| Language | Pod annotation | -|------------|-----------------------------------------------------------------------| -| Java | `admission.datadoghq.com/java-lib.version: ""` | -| Node.js | `admission.datadoghq.com/js-lib.version: ""` | -| Python | `admission.datadoghq.com/python-lib.version: ""` | -| .NET | `admission.datadoghq.com/dotnet-lib.version: ""` | -| Ruby | `admission.datadoghq.com/ruby-lib.version: ""` | -| PHP | `admission.datadoghq.com/php-lib.version: ""` | +## Remove SSI -Replace `` with the desired library version. Available versions are listed in the [Datadog container registries](#change-the-default-image-registry) and tracer source repositories for each language: +### Remove instrumentation for specific services -- [Java][34] -- [Node.js][35] -- [Python][36] -- [.NET][37] -- [Ruby][38] -- [PHP][39] +Use [workload targeting][12] (Agent v7.64+) to exclude specific services. -
Exercise caution when using the latest tag, as major library releases may introduce breaking changes.
+Alternatively, add this label to the pod spec to skip Admission Controller mutation: -For example, to automatically instrument Java applications: +
This label disables all mutating webhooks for the pod, not only SSI.
-{{< highlight yaml "hl_lines=10" >}} -apiVersion: apps/v1 -kind: Deployment -metadata: - labels: - # ... +```yaml spec: template: metadata: - annotations: - admission.datadoghq.com/java-lib.version: "" - spec: - containers: - - # ... -{{< /highlight >}} - -##### Specify at the cluster level - -If you don't enable automatic instrumentation for specific pods using annotations, you can specify which languages to instrument across the entire cluster using the SSI configuration. When `apm.instrumentation.libVersions` is set, only applications written in the specified languages are instrumented, using the specified library versions. - -The file you need to configure depends on if you enabled Single Step Instrumentation with Datadog Operator or Helm: - -{{< collapse-content title="Datadog Operator" level="h5" >}} - -For example, to instrument .NET, Python, and Node.js applications, add the following configuration to your `datadog-agent.yaml` file: - -{{< highlight yaml "hl_lines=5-8" >}} - features: - apm: - instrumentation: - enabled: true - libVersions: # Add any libraries and versions you want to set - dotnet: "x.x.x" - python: "x.x.x" - js: "x.x.x" -{{< /highlight >}} + labels: + admission.datadoghq.com/enabled: "false" +``` -{{< /collapse-content >}} +Apply the change and restart the affected pods. -{{< collapse-content title="Helm" level="h5" >}} +### Remove instrumentation for all services -For example, to instrument .NET, Python, and Node.js applications, add the following configuration to your `datadog-values.yaml` file: +{{< tabs >}} +{{% tab "Helm" %}} -{{< highlight yaml "hl_lines=5-8" >}} - datadog: - apm: - instrumentation: - enabled: true - libVersions: # Add any libraries and versions you want to set - dotnet: "x.x.x" - python: "x.x.x" - js: "x.x.x" -{{< /highlight >}} +Set `datadog.apm.instrumentation.enabled: false` in `datadog-values.yaml` and run: -{{< /collapse-content >}} +```shell +helm upgrade -f datadog-values.yaml datadog/datadog -n +``` +{{% /tab %}} +{{% tab "Datadog Operator" %}} -[34]: https://github.com/DataDog/dd-trace-java/releases -[35]: https://github.com/DataDog/dd-trace-js/releases -[36]: https://github.com/DataDog/dd-trace-py/releases -[37]: https://github.com/DataDog/dd-trace-dotnet/releases -[38]: https://github.com/DataDog/dd-trace-rb/releases -[39]: https://github.com/DataDog/dd-trace-php/releases +Set `spec.features.apm.instrumentation.enabled: false` in `datadog-agent.yaml` and apply: +```shell +kubectl apply -f datadog-agent.yaml +``` {{% /tab %}} {{< /tabs >}} -### Change the default image registry - -Datadog publishes instrumentation libraries images on gcr.io, Docker Hub, and Amazon ECR: - -| Language | gcr.io | hub.docker.com | gallery.ecr.aws | -|------------|-------------------------------------|---------------------------------------------|-------------------------------------------| -| Java | [gcr.io/datadoghq/dd-lib-java-init][15] | [hub.docker.com/r/datadog/dd-lib-java-init][16] | [gallery.ecr.aws/datadog/dd-lib-java-init][17] | -| Node.js | [gcr.io/datadoghq/dd-lib-js-init][18] | [hub.docker.com/r/datadog/dd-lib-js-init][19] | [gallery.ecr.aws/datadog/dd-lib-js-init][20] | -| Python | [gcr.io/datadoghq/dd-lib-python-init][21] | [hub.docker.com/r/datadog/dd-lib-python-init][22] | [gallery.ecr.aws/datadog/dd-lib-python-init][23] | -| .NET | [gcr.io/datadoghq/dd-lib-dotnet-init][24] | [hub.docker.com/r/datadog/dd-lib-dotnet-init][25] | [gallery.ecr.aws/datadog/dd-lib-dotnet-init][26] | -| Ruby | [gcr.io/datadoghq/dd-lib-ruby-init][27] | [hub.docker.com/r/datadog/dd-lib-ruby-init][28] | [gallery.ecr.aws/datadog/dd-lib-ruby-init][29] | -| PHP | [gcr.io/datadoghq/dd-lib-php-init][30] | [hub.docker.com/r/datadog/dd-lib-php-init][31] | [gallery.ecr.aws/datadog/dd-lib-php-init][32] | - -The `DD_ADMISSION_CONTROLLER_AUTO_INSTRUMENTATION_CONTAINER_REGISTRY` environment variable in the Datadog Cluster Agent configuration specifies the registry used by the Admission Controller. The default value is `gcr.io/datadoghq`. - -You can pull the SDK from a different registry by changing it to `docker.io/datadog`, `public.ecr.aws/datadog`, or another URL if you are hosting the images in a local container registry. - -For instructions on changing your container registry, see [Changing Your Container Registry][33]. - -### Use a private container registry +## Advanced configuration -If your organization does not allow direct pulls from public registries (such as `gcr.io`, `docker.io`, or `public.ecr.aws`), you can host the required Datadog images internally and configure the Admission Controller to use them. +The following options tune SSI behavior after the happy path is working. Each is optional. -To use SSI with a private container registry: +{{% collapse-content title="Target specific workloads" level="h3" expanded=false id="target-specific-workloads" %}} -1. Follow [these instructions][34] to mirror Datadog's container images to your private registry. +By default, SSI instruments all services in all namespaces. Use one of these methods to limit instrumentation scope. - You only need the images for the languages you are instrumenting. If you're not sure which ones you need, here's a baseline that covers most use cases: +**Namespace filtering** - - `apm-inject` - - `dd-lib-java-init` - - `dd-lib-python-init` - - `dd-lib-dotnet-init` - - `dd-lib-php-init` - - `dd-lib-ruby-init` - - `dd-lib-js-init` - - You can find these images on [gcr.io][12], [Docker Hub][13], or [Amazon ECR Public Gallery][14]. - -2. Tag the images according to your configuration. - - The versions you mirror must match the versions configured in your workloads, which might be set in one of the following ways: - - globally in the Agent config using `ddTraceVersions`, or - - per-pod using annotations like `admission.datadoghq.com/java-lib.version`. - - If no version is explicitly configured, the default version (`0`) is used. - - For example: - - ``` - apm: - instrumentation: - enabled: true - targets: - - name: "default-target" - ddTraceVersions: - java: "1" - python: "3" - ``` +Set `enabledNamespaces` or `disabledNamespaces` (mutually exclusive). Add these keys under `datadog.apm.instrumentation` (Helm) or `spec.features.apm.instrumentation` (Operator): - This configuration requires the following image tags: - - `apm-inject:0` - - `dd-lib-java-init:1` - - `dd-lib-python-init:3` - -3. Update the Cluster Agent configuration to use your private registry. +```yaml +apm: + instrumentation: + enabled: true + enabledNamespaces: + - default + - applications +``` - Set the `DD_ADMISSION_CONTROLLER_AUTO_INSTRUMENTATION_CONTAINER_REGISTRY` environment variable in your Cluster Agent config to use your private registry. +**Per-workload targeting (Agent v7.64+)** -For more details on changing your container registry, see [Changing Your Container Registry][33]. +Use `targets` to control which workloads are instrumented and what configuration they receive. Targets are evaluated in order; the first match wins. -### Using a Container Network Interface on EKS +| Key | Description | +|---------------------|-------------| +| `name` | Target block name (metadata only). | +| `namespaceSelector` | Namespace(s) to instrument. Use `matchNames`, `matchLabels`, or `matchExpressions`. See the [Kubernetes selector documentation][16]. | +| `podSelector` | Pod(s) to instrument. Use `matchLabels` or `matchExpressions`. See the [Kubernetes selector documentation][16]. | +| `ddTraceVersions` | [Datadog APM SDK][17] version per language. | +| `ddTraceConfigs` | SDK configuration: Unified Service Tags, [additional products][18], and [other APM settings][9]. | -When using a CNI like Calico, the control plane nodes are not able to initiate network connections to Datadog's Admission Controller and report an "Address is not allowed" error. -To use Single Step instrumentation, modify Datadog's Cluster Agent with the `useHostNetwork: true` parameter. +Example - instrument specific namespaces by name and label: +```yaml +apm: + instrumentation: + enabled: true + targets: + - name: login-service + namespaceSelector: + matchNames: + - login-service + ddTraceVersions: + java: "1" + ddTraceConfigs: + - name: DD_PROFILING_ENABLED + value: "auto" + - name: billing-service + namespaceSelector: + matchLabels: + app: billing-service + ddTraceVersions: + python: "4" ``` -datadog: - ... -clusterAgent: - useHostNetwork: true +Example - exclude specific pods with `matchExpressions`: - admissionController: - ... +```yaml +apm: + instrumentation: + enabled: true + targets: + - name: default-target + podSelector: + matchExpressions: + - key: app + operator: NotIn + values: + - app1 + - app2 ``` -## Remove Single Step APM instrumentation from your Agent +Example - enable additional products for a namespace: -If you don't want to collect trace data for a particular service, host, VM, or container, complete the following steps: - -### Remove instrumentation for specific services - -To remove APM instrumentation and stop sending traces from a specific service, you can do one of the following: - -#### Use instrumentation rules to target specific workloads (recommended) +```yaml +apm: + instrumentation: + enabled: true + targets: + - name: web-apps-with-security + namespaceSelector: + matchNames: + - web-apps + ddTraceVersions: + java: "1" + python: "4" + ddTraceConfigs: + - name: DD_APPSEC_ENABLED + value: "true" + - name: DD_PROFILING_ENABLED + value: "auto" +``` -With instrumentation rules (available for Agent v7.64+), you can enable and disable tracing for specific applications. [See configuration details here](#advanced-options). +**Legacy Agent (v7.63 and earlier)**: `targets` and `ddTraceVersions` are not available. Use `enabledNamespaces`/`disabledNamespaces` for namespace scope, and pin SDK versions with `libVersions` (cluster level) or the `admission.datadoghq.com/-lib.version` pod annotation. -#### Use the Datadog Admission Controller +{{% /collapse-content %}} -As an alternative, or for a version of the agent that does not support instrumentation rules, you can also disable pod mutation by adding a label to your pod. +{{% collapse-content title="Pin SDK versions" level="h3" expanded=false id="pin-sdk-versions" %}} -
In addition to disabling SSI, the following steps disable other mutating webhooks. Use with caution.
+SSI installs the latest SDK for each supported language by default. Use `ddTraceVersions` (Agent v7.64+, valid only inside a `targets` entry) to pin versions. Pinning reduces init container size, avoids downloading unnecessary SDKs, and makes tracer upgrades deliberate. -1. Set the `admission.datadoghq.com/enabled:` label to `"false"` for the pod spec: - ```yaml - spec: - template: - metadata: - labels: - admission.datadoghq.com/enabled: "false" - ``` -2. Apply the configuration: - ```shell - kubectl apply -f /path/to/your/deployment.yaml - ``` -3. Restart the services you want to remove instrumentation for. +```yaml +apm: + instrumentation: + enabled: true + targets: + - name: default-target + ddTraceVersions: + java: "1" + python: "4" + js: "5" + dotnet: "3" + ruby: "2" + php: "1" +``` -### Remove APM for all services on the infrastructure +Specify a major version (for example, `python: "4"`) to receive the latest minor release, or an exact version to pin precisely. -To stop producing traces, uninstall APM and restart the infrastructure: +{{% /collapse-content %}} -The file you need to configure depends on if you enabled Single Step Instrumentation with Datadog Operator or Helm: +{{% collapse-content title="Use opt-in labels for controlled rollout" level="h3" expanded=false %}} -{{< tabs >}} -{{% tab "Datadog Operator" %}} +| Mode | Behavior | When to use | +| ------- | ----------- | ----------- | +| Default | All supported processes are instrumented. | Small clusters or prototypes. | +| Opt-in | [Workload targeting][12] restricts instrumentation to labeled pods. | Production clusters, staged rollouts, cost-sensitive environments. | -1. Set `instrumentation.enabled=false` in `datadog-agent.yaml`: - ```yaml - features: - apm: - instrumentation: - enabled: false - ``` +Add an opt-in label to your deployment and pod template: -2. Deploy the Datadog Agent with the updated configuration file: - ```shell - kubectl apply -f /path/to/your/datadog-agent.yaml - ``` -{{% /tab %}} +```yaml +metadata: + labels: + datadoghq.com/apm-instrumentation: "enabled" +``` -{{% tab "Helm" %}} +Then configure SSI to match: -1. Set `instrumentation.enabled=false` in `datadog-values.yaml`: - ```yaml - datadog: - apm: - instrumentation: - enabled: false - ``` +```yaml +apm: + instrumentation: + enabled: true + targets: + - name: apm-instrumented + podSelector: + matchLabels: + datadoghq.com/apm-instrumentation: "enabled" +``` -2. Run the following command: - ```shell - helm upgrade datadog-agent -f datadog-values.yaml datadog/datadog - ``` -{{% /tab %}} -{{< /tabs >}} +{{% /collapse-content %}} -## Best practices +{{% collapse-content title="Configure injection modes" level="h3" expanded=false %}} -After you enable SSI, all supported processes in the cluster are automatically instrumented and begin producing traces within minutes. +SSI supports multiple injection modes that control how library files are delivered to application containers. Adjust this setting if you notice pod startup delays or high resource usage during initialization. -To control where APM is activated and reduce overhead, consider the following best practices. +| Mode | Description | Requirements | +|------|-------------|--------------| +| `auto` | Lets the Cluster Agent select the delivery method supported by the cluster. | A Cluster Agent version that supports the selected method | +| `init_container` | Copies SDK files into the pod with an init container. | Agent deployed with Helm or Operator | +| `csi` | **Experimental.** Mounts SDK files with the [Datadog CSI driver][19]. | Cluster Agent 7.76.0+ and the Datadog CSI driver | +| `image_volume` | **Experimental.** Delivers SDK files with Kubernetes image volumes, without an SDK init container. | Cluster Agent 7.77.0+ and a cluster with image-volume support | -{{% collapse-content title="Use opt-in labels for controlled APM rollout" level="h3" expanded=false id="opt-in-labels-controlled-apm-rollout" %}} +For `csi` mode, install and activate the CSI driver first. With Helm, set `datadog.csi.enabled: true`. See the [CSI driver documentation][19]. -#### Default vs. opt-in instrumentation -| Mode | Behavior | When to use | -| --- | ----------- | ----------- | -| Default | All supported processes in the cluster are instrumented. | Small clusters or prototypes. | -| Opt-in | Use [instrumentation rules][4] to restrict instrumentation to specific namespaces or pods. | Production clusters, staged rollouts, or cost‑sensitive use cases. | +**Set injection mode globally** -#### Example: Enable instrumentation for specific pods +Helm - add to `datadog-values.yaml`: +```yaml +datadog: + apm: + instrumentation: + injectionMode: +``` -1. Add a meaningful label (for example, `datadoghq.com/apm-instrumentation: "enabled"`) to both the deployment metadata and the pod template. +Datadog Operator - add to `datadog-agent.yaml`: +```yaml +features: + apm: + instrumentation: + injectionMode: +``` - ``` - apiVersion: apps/v1 - kind: Deployment - metadata: - name: checkout-api - labels: - app: checkout-api - datadoghq.com/apm-instrumentation: "enabled" # opt-in label (cluster-wide) - spec: - replicas: 3 - selector: - matchLabels: - app: checkout-api - template: - metadata: - labels: - app: checkout-api - datadoghq.com/apm-instrumentation: "enabled" # opt-in label must be on *template*, too - # Unified Service Tags (recommended) - tags.datadoghq.com/service: "checkout-api" - tags.datadoghq.com/env: "prod" - tags.datadoghq.com/version: "2025-06-10" - spec: - containers: - - name: api - image: my-registry/checkout:latest - ports: - - containerPort: 8080 - ``` +Supported values: `auto`, `init_container`, `csi`, `image_volume`. -2. In your Datadog Agent Helm config, enable SSI and use `podSelector` to inject only into pods with the matching opt-in label. +**Set injection mode per pod** - ``` - apm: - instrumentation: - enabled: true - targets: - - name: apm-instrumented - podSelector: - matchLabels: - datadoghq.com/apm-instrumentation: "enabled" - ``` +Add this annotation to the pod spec: +```yaml +metadata: + annotations: + admission.datadoghq.com/apm-inject.injection-mode: "" +``` -See [instrumentation rules][4] for additional examples. +Supported values: `auto`, `init_container`, `csi`, `image_volume`. {{% /collapse-content %}} +{{% collapse-content title="Change the image registry" level="h3" expanded=false %}} -{{% collapse-content title="Control which Datadog SDKs are loaded" level="h3" expanded=false id="control-loaded-datadog-sdks" %}} +Datadog publishes SDK images to multiple public registries. The default can vary by Datadog site and cluster environment. For the current registry list and configuration options, see [Changing Your Container Registry][20]. -Use `ddTraceVersions` in your Agent Helm config to control both the language and the version of the Datadog SDK. This prevents unnecessary SDKs from being downloaded, which minimizes init-container footprint, reduces image size, and allows for more deliberate tracer upgrades (for example, to meet compliance requirements or simplify debugging). +**Use a private container registry** -#### Example: Specify a Java SDK for a namespace +If your organization cannot pull from public registries, mirror the Datadog images to your private registry: -Only Java applications run in the `login-service` namespace. To avoid downloading other SDKs, configure the Agent to target that namespace and inject only the Java SDK version 1.48.2. +1. [Mirror the images][21] for the languages you are instrumenting. At minimum, mirror `apm-inject` and the `dd-lib--init` images required by your configured injection mode. +2. Tag the images to match your configuration. If you set `ddTraceVersions` to `java: "1"` and `python: "4"`, mirror: + - `apm-inject:0` + - `dd-lib-java-init:1` + - `dd-lib-python-init:4` -``` -targets: - - name: login-service - namespaceSelector: - matchNames: ["login-service"] - ddTraceVersions: - java: "1.48.2" # pin version -``` +3. Set `DD_ADMISSION_CONTROLLER_AUTO_INSTRUMENTATION_CONTAINER_REGISTRY` to your private registry URL. + +{{% /collapse-content %}} -#### Default configuration +{{% collapse-content title="Container Network Interface on EKS" level="h3" expanded=false %}} -If a pod doesn't match any `ddTraceVersions` rule, the default target applies. +When using a CNI like Calico on EKS, control plane nodes cannot connect to the Admission Controller. Set `useHostNetwork: true` on the Cluster Agent: -``` -targets: - - name: default-target # tag any pod *without* an override - ddTraceVersions: - java: "1" # stay on latest v1.x - python: "3" # stay on latest v3.x - js: "5" # NodeJS - php: "1" - dotnet: "3" +```yaml +clusterAgent: + useHostNetwork: true ``` {{% /collapse-content %}} ## Troubleshooting -If you encounter problems enabling APM with SSI, see the [SSI troubleshooting guide][35]. +If you encounter problems with SSI, see the [SSI troubleshooting guide][8]. ## Further reading @@ -864,35 +666,21 @@ If you encounter problems enabling APM with SSI, see the [SSI troubleshooting gu [1]: https://v3.helm.sh/docs/intro/install/ [2]: https://kubernetes.io/docs/tasks/tools/install-kubectl/ -[3]: /tracing/glossary/#instrumentation -[4]: /tracing/trace_collection/automatic_instrumentation/single-step-apm/kubernetes/?tab=agentv764recommended#configure-instrumentation-for-namespaces-and-pods +[3]: /tracing/trace_collection/single-step-apm/compatibility/ [5]: /getting_started/tagging/unified_service_tagging/?tab=kubernetes#containerized-environment -[7]: /tracing/trace_collection/library_config/ -[11]: https://app.datadoghq.com/fleet/install-agent/latest?platform=kubernetes -[12]: https://gcr.io/datadoghq -[13]: https://hub.docker.com/u/datadog -[14]: https://gallery.ecr.aws/datadog -[15]: http://gcr.io/datadoghq/dd-lib-java-init -[16]: http://hub.docker.com/r/datadog/dd-lib-java-init -[17]: http://gallery.ecr.aws/datadog/dd-lib-java-init -[18]: http://gcr.io/datadoghq/dd-lib-js-init -[19]: http://hub.docker.com/r/datadog/dd-lib-js-init -[20]: http://gallery.ecr.aws/datadog/dd-lib-js-init -[21]: http://gcr.io/datadoghq/dd-lib-python-init -[22]: http://hub.docker.com/r/datadog/dd-lib-python-init -[23]: http://gallery.ecr.aws/datadog/dd-lib-python-init -[24]: http://gcr.io/datadoghq/dd-lib-dotnet-init -[25]: http://hub.docker.com/r/datadog/dd-lib-dotnet-init -[26]: http://gallery.ecr.aws/datadog/dd-lib-dotnet-init -[27]: http://gcr.io/datadoghq/dd-lib-ruby-init -[28]: http://hub.docker.com/r/datadog/dd-lib-ruby-init -[29]: http://gallery.ecr.aws/datadog/dd-lib-ruby-init -[30]: http://gcr.io/datadoghq/dd-lib-php-init -[31]: http://hub.docker.com/r/datadog/dd-lib-php-init -[32]: http://gallery.ecr.aws/datadog/dd-lib-php-init -[33]: /containers/guide/changing_container_registry/ -[34]: /containers/guide/sync_container_images/#copy-an-image-to-another-registry-using-crane -[35]: /tracing/trace_collection/automatic_instrumentation/single-step-apm/troubleshooting -[36]: /tracing/trace_collection/automatic_instrumentation/single-step-apm/compatibility/ -[37]: /containers/kubernetes/csi_driver/ -[41]: /tracing/guide/injectors/ +[6]: https://app.datadoghq.com/apm/services +[7]: https://app.datadoghq.com/apm/traces +[8]: /tracing/trace_collection/single-step-apm/troubleshooting/ +[9]: /tracing/trace_collection/library_config/ +[10]: https://app.datadoghq.com/organization-settings/api-keys +[11]: /getting_started/site/ +[12]: #target-specific-workloads +[13]: #pin-sdk-versions +[14]: https://app.datadoghq.com/fleet/install-agent/latest?platform=kubernetes +[15]: #configure-injection-modes +[16]: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#resources-that-support-set-based-requirements +[17]: /tracing/trace_collection/single-step-apm/compatibility/#supported-language-runtimes +[18]: #enable-additional-products +[19]: /containers/kubernetes/csi_driver/ +[20]: /containers/guide/changing_container_registry/ +[21]: /containers/guide/sync_container_images/#copy-an-image-to-another-registry-using-crane diff --git a/content/en/tracing/trace_collection/single-step-apm/linux.md b/content/en/tracing/trace_collection/single-step-apm/linux.md index 1c05450ef09..fa3090f3b7f 100644 --- a/content/en/tracing/trace_collection/single-step-apm/linux.md +++ b/content/en/tracing/trace_collection/single-step-apm/linux.md @@ -1,5 +1,6 @@ --- title: Single Step APM Instrumentation on Linux +description: "Enable Datadog Single Step Instrumentation on a Linux host or VM, verify traces, configure SDK versions and instrumentation rules, and remove instrumentation." code_lang: linux type: multi-code-lang code_lang_weight: 0 @@ -13,95 +14,61 @@ further_reading: ## Overview -On a Linux host or VM, use Single Step Instrumentation (SSI) for APM to install the Datadog Agent and [instrument][14] your applications in one step, with no additional configuration required. +On a Linux host or VM, Single Step Instrumentation (SSI) installs the Datadog Agent and [instruments][1] your applications in one step, with no additional configuration required. -## Enable APM on your applications +Before you start, confirm that your environment is supported in the [SSI compatibility guide][2]. -
Before proceeding, confirm that your environment is compatible by reviewing the SSI compatibility guide.
+## Enable APM on your applications ### New Agent installation -If you don't yet have a Datadog Agent installed, follow these steps to install the Agent and enable SSI simultaneously. - -1. In Datadog, go to the [Install the Datadog Agent on Linux][15] page. -2. In the {{< ui >}}Customize your Agent coverage{{< /ui >}} section, go to {{< ui >}}Core Observability{{< /ui >}} and turn on {{< ui >}}Application Performance Monitoring{{< /ui >}}. - - {{< img src="tracing/trace_collection/enable_apm.png" alt="The 'Customize your Agent coverage' section of in-app instructions for installing the Datadog Agent on Linux" style="width:100%;" >}} - -{{< site-region region="us,us3,us5,eu,ap1,ap2,uk1" >}} -3. (Optional) By default, SSI installs the latest SDK versions. To use specific versions instead: +If you don't have a Datadog Agent installed, run the following command on your Linux host or VM to install the Agent and enable SSI at the same time: - Click {{< ui >}}Customize Library Versions{{< /ui >}}, then select your desired version for each language from the dropdowns. You can select an exact version or a major version, which uses the latest minor release available when the installation command is run. +```shell +DD_API_KEY= DD_SITE="{{< region-param key="dd_site" >}}" DD_APM_INSTRUMENTATION_ENABLED=host \ + bash -c "$(curl -L https://install.datadoghq.com/scripts/install_script_agent7.sh)" +``` - {{< img src="tracing/trace_collection/customize_library_versions.png" alt="The 'Customize library versions' drop-down in the instructions for installing the Datadog Agent on Linux" style="width:100%;" >}} - - Available versions are listed in source repositories for each language: [Java][8] (`java`), [Node.js][9] (`js`), [Python][10] (`python`), [.NET][11] (`dotnet`), [Ruby][12] (`ruby`), [PHP][13] (`php`). +Replace `` with your [Datadog API key][3]. After the command completes, restart your applications. -[8]: https://github.com/DataDog/dd-trace-java/releases -[9]: https://github.com/DataDog/dd-trace-js/releases -[10]: https://github.com/DataDog/dd-trace-py/releases -[11]: https://github.com/DataDog/dd-trace-dotnet/releases -[12]: https://github.com/DataDog/dd-trace-rb/releases -[13]: https://github.com/DataDog/dd-trace-php/releases +By default, SSI installs the latest SDK versions for all supported languages. -{{< /site-region >}} - -{{< site-region region="gov" >}} +#### Optional: pin SDK versions -3. (Optional) By default, SSI installs the latest SDK versions. To use specific versions instead, set your desired library versions with the `DD_APM_INSTRUMENTATION_LIBRARIES` variable in your Agent installation command: +To install specific SDK versions instead of the latest, add the `DD_APM_INSTRUMENTATION_LIBRARIES` variable: - ```shell - DD_API_KEY= - DD_SITE="US1-FED" - DD_APM_INSTRUMENTATION_ENABLED=host - DD_APM_INSTRUMENTATION_LIBRARIES="java:1,python:2,js:5,dotnet:3,php:1" - bash -c "$(curl -L https://install.datadoghq.com/scripts/install_script_agent7.sh)" - ``` +```shell +DD_API_KEY= DD_SITE="{{< region-param key="dd_site" >}}" DD_APM_INSTRUMENTATION_ENABLED=host \ + DD_APM_INSTRUMENTATION_LIBRARIES="java:1,python:4,js:5,dotnet:3,ruby:2,php:1" \ + bash -c "$(curl -L https://install.datadoghq.com/scripts/install_script_agent7.sh)" +``` - Available versions are listed in source repositories for each language: [Java][8] (`java`), [Node.js][9] (`js`), [Python][10] (`python`), [.NET][11] (`dotnet`), [Ruby][12] (`ruby`), [PHP][13] (`php`). +You can specify an exact version or a major version, which uses the latest minor release available when the command runs. Available versions are listed in the source repository for each language: [Java][4] (`java`), [Node.js][5] (`js`), [Python][6] (`python`), [.NET][7] (`dotnet`), [Ruby][8] (`ruby`), and [PHP][9] (`php`). -[8]: https://github.com/DataDog/dd-trace-java/releases -[9]: https://github.com/DataDog/dd-trace-js/releases -[10]: https://github.com/DataDog/dd-trace-py/releases -[11]: https://github.com/DataDog/dd-trace-dotnet/releases -[12]: https://github.com/DataDog/dd-trace-rb/releases -[13]: https://github.com/DataDog/dd-trace-php/releases +
SSI adds a small amount of startup time to instrumented applications. If this overhead is not acceptable for your use case, contact Datadog Support.
-{{< /site-region >}} +{{% collapse-content title="Alternate: install with the in-app wizard" level="h4" expanded=false %}} -{{< site-region region="gov2" >}} +To generate an installation command from the Datadog UI: -3. (Optional) By default, SSI installs the latest SDK versions. To use specific versions instead, set your desired library versions with the `DD_APM_INSTRUMENTATION_LIBRARIES` variable in your Agent installation command: +1. Go to the [Install the Datadog Agent on Linux][10] page. +1. In the {{< ui >}}Customize your Agent coverage{{< /ui >}} section, go to {{< ui >}}Core Observability{{< /ui >}} and turn on {{< ui >}}Application Performance Monitoring{{< /ui >}}. - ```shell - DD_API_KEY= - DD_SITE="US2-FED" - DD_APM_INSTRUMENTATION_ENABLED=host - DD_APM_INSTRUMENTATION_LIBRARIES="java:1,python:2,js:5,dotnet:3,php:1" - bash -c "$(curl -L https://install.datadoghq.com/scripts/install_script_agent7.sh)" - ``` + {{< img src="tracing/trace_collection/enable_apm.png" alt="The 'Customize your Agent coverage' section of in-app instructions for installing the Datadog Agent on Linux" style="width:100%;" >}} - Available versions are listed in source repositories for each language: [Java][8] (`java`), [Node.js][9] (`js`), [Python][10] (`python`), [.NET][11] (`dotnet`), [Ruby][12] (`ruby`), [PHP][13] (`php`). +1. (Optional) To pin SDK versions, click {{< ui >}}Customize Library Versions{{< /ui >}}, then select a version for each language from the dropdowns. -[8]: https://github.com/DataDog/dd-trace-java/releases -[9]: https://github.com/DataDog/dd-trace-js/releases -[10]: https://github.com/DataDog/dd-trace-py/releases -[11]: https://github.com/DataDog/dd-trace-dotnet/releases -[12]: https://github.com/DataDog/dd-trace-rb/releases -[13]: https://github.com/DataDog/dd-trace-php/releases + {{< img src="tracing/trace_collection/customize_library_versions.png" alt="The 'Customize library versions' drop-down in the instructions for installing the Datadog Agent on Linux" style="width:100%;" >}} -{{< /site-region >}} +1. Copy and run the generated command on your Linux host or VM, then restart your applications. -4. Copy and run the Agent installation command on your Linux host or VM. -5. Restart your applications. - -
SSI adds a small amount of startup time to instrumented applications. If this overhead is not acceptable for your use case, contact Datadog Support.
+{{% /collapse-content %}} ### Existing Agent installation -If you already have a Datadog Agent installed, use Fleet Automation to enable SSI. +If you already have a Datadog Agent installed, use Fleet Automation to enable SSI: -1. In Datadog, go to [**Fleet Automation > Configuration**][21]. +1. In Datadog, go to [**Fleet Automation > Configuration**][11]. 1. Click {{< ui >}}Configure Agents{{< /ui >}}. 1. Apply filters to select the agents you want to configure, then click **Next**. @@ -118,9 +85,40 @@ If you already have a Datadog Agent installed, use Fleet Automation to enable SS 1. Click **Next**. 1. Review your configuration and click {{< ui >}}Deploy Configuration{{< /ui >}}. +## Verify your first trace + +After you enable SSI, confirm that your applications are instrumented and sending traces: + +1. Restart your instrumented applications, then send requests to generate traffic. +1. Confirm that the SSI launcher is loaded into a running process. Replace `` with the process ID of an instrumented application: + + ```shell + cat /proc//maps + ``` + + The output includes `launcher.preload.so`. This confirms that the injector loaded; the tracer itself may be loaded through language-specific arguments or environment variables rather than as a mapped shared library. See [per-runtime instrumentation][17] for those mechanisms. + +1. Confirm that the launcher is registered for preloading on the host: + + ```shell + cat /etc/ld.so.preload + ``` + + The output includes the path to `launcher.preload.so`. + +1. Check the APM section of the Agent status: + + ```shell + datadog-agent status + ``` + +1. In Datadog, go to [**APM > Catalog**][18] and confirm that your services appear. + +If your traces don't appear, see the [SSI troubleshooting guide][14]. + ## Configure Unified Service Tags -Unified Service Tags (USTs) apply consistent tags across traces, metrics, and logs, making it easier to navigate and correlate your observability data. Learn how to [set USTs for Linux services][16]. +Unified Service Tags (USTs) apply consistent tags across traces, metrics, and logs, making it easier to navigate and correlate your observability data. See [Set USTs for Linux services][12]. ## Enable SDK-dependent products and features @@ -130,15 +128,15 @@ After SSI loads the Datadog SDK into your applications and enables distributed t Use one of the following setup methods: -- **[Configure in `application_monitoring.yaml`][18]**: +- **[Configure in `application_monitoring.yaml`][13]**: Configure products and features across all services on a host without modifying application command lines. -- **[Set environment variables][17]**: +- **[Set environment variables][15]**: - Enable products by setting environment variables directly in your application configuration. + Enable products by setting environment variables directly in your application configuration. -## Advanced options +## Advanced configuration ### Update SDK version @@ -152,19 +150,19 @@ To update the SDK versions: ### Define instrumentation rules {{< site-region region="gov" >}} -
Instrumentation rules are not supported for your selected Datadog site ({{< region-param key="dd_site_name" >}}).
+
Instrumentation rules are not supported for your selected Datadog site ({{< region-param key="dd_site_name" >}}).
{{< /site-region >}} Instrumentation rules (available for Agent v7.73+) let you control which processes are automatically instrumented by SSI on Linux hosts. To configure instrumentation rules: -1. In Datadog, go to {{< ui >}}APM{{< /ui >}} > {{< ui >}}Service Setup{{< /ui >}} > [{{< ui >}}Manage Instrumentation Rules{{< /ui >}}][20]. +1. In Datadog, go to {{< ui >}}APM{{< /ui >}} > {{< ui >}}Service Setup{{< /ui >}} > [{{< ui >}}Manage Instrumentation Rules{{< /ui >}}][16]. 1. Click {{< ui >}}Add or Edit Rules{{< /ui >}}. 1. Define instrumentation rules: 1. Click {{< ui >}}Add New Rule{{< /ui >}}, then choose {{< ui >}}Allow Rule{{< /ui >}} or {{< ui >}}Block Rule{{< /ui >}} to specify whether matching processes should be instrumented. 1. Name your rule. - 1. Add one or more conditions. See [Define rule conditions](#define-rule-conditions) to learn more. + 1. Add one or more conditions. See [Define rule conditions][19] to learn more. {{< img src="tracing/trace_collection/define_instrumentation_rule.png" alt="The instrumentation rules UI, showing configuration options for defining a rule" style="width:100%;" >}} @@ -230,23 +228,28 @@ To stop producing traces for all services on your infrastructure: ## Troubleshooting -If you encounter problems enabling APM with SSI, see the [SSI troubleshooting guide][19]. +If you encounter problems enabling APM with SSI, see the [SSI troubleshooting guide][14]. ## Further reading {{< partial name="whats-next/whats-next.html" >}} -[8]: https://github.com/DataDog/dd-trace-java/releases -[9]: https://github.com/DataDog/dd-trace-js/releases -[10]: https://github.com/DataDog/dd-trace-py/releases -[11]: https://github.com/DataDog/dd-trace-dotnet/releases -[12]: https://github.com/DataDog/dd-trace-rb/releases -[13]: https://github.com/DataDog/dd-trace-php/releases -[14]: /tracing/glossary/#instrumentation -[15]: https://app.datadoghq.com/fleet/install-agent/latest?platform=linux -[16]: /getting_started/tagging/unified_service_tagging/?tab=kubernetes#non-containerized-environment -[17]: /tracing/trace_collection/library_config/ -[18]: /tracing/trace_collection/library_config/application_monitoring_yaml/ -[19]: /tracing/trace_collection/automatic_instrumentation/single-step-apm/troubleshooting -[20]: https://app.datadoghq.com/apm/service-setup/workload-selection -[21]: https://app.datadoghq.com/fleet/agent-management +[1]: /tracing/glossary/#instrumentation +[2]: /tracing/trace_collection/single-step-apm/compatibility/ +[3]: https://app.datadoghq.com/organization-settings/api-keys +[4]: https://github.com/DataDog/dd-trace-java/releases +[5]: https://github.com/DataDog/dd-trace-js/releases +[6]: https://github.com/DataDog/dd-trace-py/releases +[7]: https://github.com/DataDog/dd-trace-dotnet/releases +[8]: https://github.com/DataDog/dd-trace-rb/releases +[9]: https://github.com/DataDog/dd-trace-php/releases +[10]: https://app.datadoghq.com/fleet/install-agent/latest?platform=linux +[11]: https://app.datadoghq.com/fleet/agent-management +[12]: /getting_started/tagging/unified_service_tagging/#non-containerized-environment +[13]: /tracing/trace_collection/library_config/application_monitoring_yaml/ +[14]: /tracing/trace_collection/single-step-apm/troubleshooting/ +[15]: /tracing/trace_collection/library_config/ +[16]: https://app.datadoghq.com/apm/service-setup/workload-selection +[17]: /tracing/guide/injectors/#per-runtime-instrumentation +[18]: https://app.datadoghq.com/services +[19]: #define-rule-conditions diff --git a/content/en/tracing/trace_collection/single-step-apm/troubleshooting.md b/content/en/tracing/trace_collection/single-step-apm/troubleshooting.md index b5a9c914de8..54aab2416b3 100644 --- a/content/en/tracing/trace_collection/single-step-apm/troubleshooting.md +++ b/content/en/tracing/trace_collection/single-step-apm/troubleshooting.md @@ -1,9 +1,10 @@ --- title: Troubleshooting Single Step APM +description: "Diagnose and fix Single Step Instrumentation (SSI) issues on Kubernetes, Linux, Docker, and Windows. Covers missing traces, silent failure modes, injection verification, Fleet Automation diagnostics, and language-specific troubleshooting." aliases: - /tracing/trace_collection/automatic_instrumentation/single-step-apm/troubleshooting/ further_reading: -- link: /tracing/trace_collection/automatic_instrumentation/single-step-apm/ +- link: /tracing/trace_collection/single-step-apm/ tag: Documentation text: Single Step APM Instrumentation - link: https://learn.datadoghq.com/courses/troubleshooting-apm-instrumentation-on-a-host @@ -13,16 +14,204 @@ further_reading: ## Overview -Single Step Instrumentation (SSI) helps instrument applications by automatically loading application processes with the Datadog SDKs. SSI works for applications running on Linux hosts, in container environments such as Kubernetes and Docker, and for .NET applications served by Windows IIS—without requiring changes to application dependencies or images. If you encounter issues enabling APM with SSI, use this guide to troubleshoot and resolve common problems. For further assistance, contact [Datadog Support][1]. +[Single Step Instrumentation (SSI)][2] instruments applications by automatically loading application processes with the Datadog SDKs. SSI works for applications running on Linux hosts, in container environments such as Kubernetes and Docker, and for .NET applications served by Windows IIS, without requiring changes to application dependencies or images. If you enabled SSI but don't see traces, use this guide to find and fix the cause. For further assistance, contact [Datadog Support][1]. + +## How SSI injection works + +Understanding the injection mechanism helps you reason about where a failure occurs. + +**Kubernetes:** + +1. An admission webhook (registered by the Cluster Agent) intercepts pod creation. +2. The webhook mutates the pod spec to deliver the injector and SDKs. Depending on `injectionMode`, delivery uses init containers, the Datadog CSI driver, or Kubernetes image volumes. +3. The webhook configures the application container so the injector loads when the application process starts. +4. The injector detects the runtime and configures the matching tracer SDK. + +**Linux hosts:** + +1. The installer adds the Datadog launcher to `/etc/ld.so.preload`. +2. Each newly launched process preloads `launcher.preload.so`. +3. The launcher detects the runtime and configures the matching tracer SDK. + +**Docker:** + +1. The installer configures Docker to use the Datadog `runc` shim. +2. When a new application container starts, the shim mounts the injector and SDK files and sets `LD_PRELOAD` in the container. +3. The launcher detects the runtime and configures the matching tracer SDK. + +Two vantage points help you locate a failure: + +- Cluster, host, or container state shows whether injection was applied. +- The Datadog UI (**APM** > **Catalog** and **Traces**) shows what the backend received. + +If injection was applied but no traces reach the backend, the problem is post-injection: the tracer isn't reporting, the Agent can't be reached, or existing instrumentation took precedence. + +SSI fails silently in several cases. It produces no error when it detects existing instrumentation, when the runtime version is unsupported, when a process isn't restarted after SSI is enabled, or when a pod is opted out or runs in the Agent's namespace. The checks below surface these cases. + +## First checks: no traces after enabling SSI + +Work through these checks in order for your platform. + +### All platforms + +1. **Did you restart your application after enabling SSI?** SSI injects at startup. Existing processes and pods are not instrumented until restarted. + +2. **Does your application already load a tracing SDK?** Check dependency manifests, lock files, Dockerfiles, and startup scripts: + ```shell + grep -RniE "ddtrace|dd-trace|dd-java-agent|javaagent|opentelemetry" \ + --include='requirements*.txt' --include='pyproject.toml' --include='poetry.lock' \ + --include='package*.json' --include='yarn.lock' --include='pnpm-lock.yaml' \ + --include='Gemfile*' --include='pom.xml' --include='build.gradle*' \ + --include='*.csproj' --include='Dockerfile*' . + ``` + For Java, also check for `-javaagent` flags and inspect `JAVA_TOOL_OPTIONS`. Review each match: remove or disable only conflicting tracer initialization and dependencies, then rebuild the application. Do not remove OpenTelemetry API, metrics, or logging dependencies solely because they matched the search. + +3. **Is the runtime version supported?** Check the [SSI compatibility guide][3]. + +### Kubernetes + +First, determine which delivery method the pod uses. Inspect the pod annotation and your Helm or Operator configuration for `injectionMode`: + +```shell +kubectl get pod -n -o jsonpath='{.metadata.annotations.admission\.datadoghq\.com/apm-inject\.injection-mode}' +``` + +If the value is empty or `auto`, inspect the mutated pod to see which method the Cluster Agent selected: + +```shell +kubectl describe pod -n +``` + +- With `init_container`, the pod has a `datadog-lib--init` init container. +- With `csi` or `image_volume`, no SDK init container is expected; look for Datadog-injected volumes and mounts and check pod events for mount errors. + +#### Pod was not mutated + +Injection was never applied to the pod. Check: + +- **Namespace and target configuration:** + + For the Datadog Operator: + + ```shell + kubectl get datadogagent -n -o yaml | grep -A 15 instrumentation + ``` + + For Helm: + + ```shell + helm get values -n | grep -A 15 instrumentation + ``` + + Confirm your `enabledNamespaces`, `disabledNamespaces`, or `podSelector` targets match your application's namespace and labels. +- **Opt-out label**: if this returns `false`, the Admission Controller skips the pod: + ```shell + kubectl get pod -n -o jsonpath='{.metadata.labels.admission\.datadoghq\.com/enabled}' + ``` +- **Pod labels** (confirm the pod matches your targeting selectors): + ```shell + kubectl get pod -n --show-labels + ``` +- **Admission webhook registered:** + ```shell + kubectl get mutatingwebhookconfigurations | grep datadog + ``` +- **Cluster Agent running:** + ```shell + kubectl get pods -n -l app=datadog-cluster-agent + ``` +- **Agent namespace**: SSI does not instrument pods in the namespace where the Datadog Agent runs. + +#### Pod was mutated but no traces arrive + +The library was injected but no traces arrive. Check: + +- **Existing instrumentation** in application code or dependency manifests. Manual instrumentation (a bundled tracer, `-javaagent`, or `ddtrace` imports) causes SSI to disable itself silently. +- **Agent APM receiver:** + ```shell + kubectl exec -n $(kubectl get pod -n -l app=datadog-agent -o name | head -1) -- agent status | grep -A 5 "APM Agent" + ``` +- **Cluster node the pod is scheduled on** (for correlating Agent connectivity): + ```shell + kubectl get pod -n -o jsonpath='{.spec.nodeName}' + ``` +- **Wrong service or environment** on the traces you do see indicates a Unified Service Tagging problem. See [Unified Service Tagging][7] and [Injection appears successful but traces are missing][16]. + +#### Runtime version (when compatibility is suspected) + +```shell +kubectl exec -n -- python --version +kubectl exec -n -- node --version +kubectl exec -n -- java -version +``` + +Compare against the [SSI compatibility guide][3]. + +### Linux hosts + +1. **Is the launcher loaded into the process?** Check the process memory map: + ```shell + grep "launcher.preload.so" /proc//maps + ``` + A match confirms that the launcher loaded. It does not, by itself, prove that the language tracer loaded; tracers can be configured through runtime arguments or environment variables. Use [injector debug logs][17] or inspect the [language-specific injection setting][18] to confirm that step. + +2. **Was the process started after SSI was enabled?** The preload applies only to processes launched after `/etc/ld.so.preload` was written. Compare the process start time against the file's modification time: + ```shell + ps -o lstart= -p + stat -c %y /etc/ld.so.preload + ``` + +3. **Is `/etc/ld.so.preload` configured?** SSI on Linux uses `/etc/ld.so.preload` to load the launcher. Check: + ```shell + cat /etc/ld.so.preload + ``` + The output should contain `/opt/datadog-packages/datadog-apm-inject/stable/inject/launcher.preload.so`. + +4. **Is the application using musl libc or a static binary?** The SSI launcher on Linux requires glibc. Applications on Alpine or other musl-based distributions, and statically linked binaries (common with Go), cannot be instrumented with SSI. Check the libc and binary type: + ```shell + ldd --version # musl indicates an incompatible libc + file /proc//exe # "statically linked" indicates an incompatible binary + ``` + +5. **Is SELinux or AppArmor blocking the launcher?** Security policies can prevent `/etc/ld.so.preload` from being read. Check `getenforce` (SELinux) or `dmesg | grep apparmor` (AppArmor) for denials. + +6. **Is the Agent APM receiver up?** + ```shell + datadog-agent status | grep -A 5 "APM Agent" + ``` + +If none of these apply, continue with the diagnostic methods below. + +### Docker + +Docker injection uses a custom runtime shim and container-level `LD_PRELOAD`; it does not depend on the host's `/etc/ld.so.preload`. + +1. **Was the application container created after Docker SSI was enabled?** Recreate or restart the container so the Datadog runtime shim can inject it. +2. **Is the Agent container configured for SSI?** Confirm that it has APM enabled, uses `/opt/datadog/apm/inject/run/apm.socket`, and shares `/opt/datadog/apm` with the host. +3. **Was the injector added to the application process?** Inspect the process environment for `LD_PRELOAD` and the setting used by its language runtime: + + ```shell + docker exec sh -c 'tr "\0" "\n" < /proc/1/environ | grep -E "LD_PRELOAD|JAVA_TOOL_OPTIONS|NODE_OPTIONS|PYTHONPATH|CORECLR_"' + ``` + +4. **Is the Agent APM receiver up?** Run `agent status` in the Agent container and check the APM section: + + ```shell + docker exec agent status + ``` + +5. **What decision did the injector make?** Enable `DD_APM_INSTRUMENTATION_DEBUG=true` on the application container, recreate it, and inspect `docker logs `. + +For rootless Docker or custom runtime shims, also review the Docker limitations in the [SSI compatibility guide][3]. ## Troubleshooting methods -You can investigate injection issues in the Datadog UI with Fleet Automation, or manually at the container level. For a lower-level view of what the injector decided for each process, see [Injector debug logs](#injector-debug-logs). +You can investigate injection issues in the Datadog UI with Fleet Automation, or manually at the container level. For a lower-level view of what the injector decided for each process, see [Injector debug logs][17]. ### Troubleshoot injection in Datadog Fleet Automation - Fleet Automation provides two types of instrumentation insights for SSI: + - **Process-level insights** show instrumentation status and SDK installation details for individual hosts or containers. - **Kubernetes cluster insights** provide a higher-level view of instrumentation across your clusters, helping you understand how SSI configuration and injection are applied at scale. @@ -36,11 +225,11 @@ This functionality is available for: - **Environments**: Linux hosts, containers, Kubernetes - Datadog Agent v7.68.2+ -#### View process-level insights +#### View process-level insights Use process-level insights to verify whether SSI has been correctly applied to your application processes and to identify any injection failures. -1. Navigate to [Fleet Automation][9]. +1. Navigate to [Fleet Automation][5]. 1. Use facets to filter down to relevant hosts: - `single_step_instrumentation` shows which hosts have SSI enabled or disabled. - `single_step_instrumentation_status` shows which hosts have encountered issues with service instrumentation. @@ -54,7 +243,7 @@ Use process-level insights to verify whether SSI has been correctly applied to y Use cluster-level insights to understand how SSI is configured and functioning across your Kubernetes clusters. These insights extend troubleshooting beyond individual processes to show how instrumentation is applied to workloads at the cluster level. -1. Navigate to [{{< ui >}}Fleet Automation{{< /ui >}}][9] > {{< ui >}}View Agents{{< /ui >}}, and select {{< ui >}}Kubernetes Clusters{{< /ui >}} in the upper-right corner. +1. Navigate to [{{< ui >}}Fleet Automation{{< /ui >}}][5] > {{< ui >}}View Agents{{< /ui >}}, and select {{< ui >}}Kubernetes Clusters{{< /ui >}} in the upper-right corner. 1. Select a cluster to view its details, including: - Whether the cluster is managed by Helm or the Datadog Operator - The Cluster Agent and Node Agent versions @@ -64,29 +253,25 @@ Use cluster-level insights to understand how SSI is configured and functioning a - The pods identified as instrumentation targets based on cluster configuration or pod-level annotations - The status of each targeted pod, including whether instrumentation succeeded - The SDKs injected into each pod, including language and version - - Whether each instrumented workload is generating traces -1. Hover over any status icon to see contextual details about the state of instrumentation or trace collection. + - Whether each instrumented workload is generating traces +1. Hover over any status icon to see contextual details about the state of instrumentation or trace collection. {{< img src="tracing/trace_collection/k8s-ssi-tab.png" alt="The Single Step Instrumentation tab for a Kubernetes cluster, showing the SSI config yaml and a list of instrumented pods" style="width:100%;" >}} -### Manually verify injection in the application container +### Manually verify injection -If the Datadog UI does not show any instrumentation issues, or if you're troubleshooting a single service or container, you can manually verify whether injection occurred as expected. This method is helpful when debugging in environments where centralized visibility is limited or when a specific service isn't reporting traces. +If the Datadog UI does not show any instrumentation issues, manually verify the stages that apply to your environment: -To confirm injection at the container level, check that: +- **Linux host:** Confirm that `/etc/ld.so.preload` contains `/opt/datadog-packages/datadog-apm-inject/stable/inject/launcher.preload.so` and that the injector package directory exists. +- **Docker:** Confirm that the application process has an `LD_PRELOAD` setting for the Datadog launcher and that the Agent and application containers can access the injected files and APM socket under `/opt/datadog/apm`. +- **Kubernetes:** Inspect the mutated pod. Init-container mode adds `datadog-lib--init`; CSI and image-volume modes add Datadog volumes and mounts without that SDK init container. -1. `/etc/ld.so.preload` includes the following entry: - ``` - /opt/datadog-packages/datadog-apm-inject/stable/inject/launcher.preload.so - ``` -2. The `LD_PRELOAD` environment variable is set to the same value. -3. The directory `/opt/datadog-packages/datadog-apm-inject` exists, with `stable` and `$version` subdirectories. -4. Language-specific directories exist (for example, `/opt/datadog/apm/library/java/` for Java). +On Linux-based platforms, finding `launcher.preload.so` in `/proc//maps` confirms that the injector loaded into that process. Confirm tracer configuration separately by checking the runtime-specific argument or environment variable in the [injector guide][18], or by reviewing injector debug logs. To enable debug logs during manual verification: 1. Set the following in your pod spec: - + {{< code-block lang="yaml" disable_copy="true" collapsible="true" >}} env: - name: DD_TRACE_DEBUG # debug logging for the SDK @@ -94,15 +279,14 @@ To enable debug logs during manual verification: - name: DD_APM_INSTRUMENTATION_DEBUG # debug logging for the injector value: "true" {{< /code-block >}} - -2. Delete the pod to enable debug logs during injection. +2. Delete the pod to enable debug logs during injection. ## Injector debug logs -Injector debug logs show what the injector decided for each process: whether injection succeeded, was denied, or was skipped, and why. Enable them when [Fleet Automation](#troubleshoot-injection-in-datadog-fleet-automation) doesn't explain a failure, when you're diagnosing at the host or container level, or when you're collecting information for [Datadog Support][1]. +Injector debug logs show what the injector decided for each process: whether injection succeeded, was denied, or was skipped, and why. Enable them when [Fleet Automation][19] doesn't explain a failure, when you're diagnosing at the host or container level, or when you're collecting information for [Datadog Support][1]. -Injector debug logs are separate from tracer debug logs. Use injector logs to diagnose whether and how a tracer was injected; after injection, use [tracer debug logs][13] to diagnose the tracer running inside the process. +Injector debug logs are separate from tracer debug logs. Use injector logs to diagnose whether and how a tracer was injected; after injection, use [tracer debug logs][6] to diagnose the tracer running inside the process. ### Enable debug mode @@ -241,9 +425,9 @@ A policy prevented injection for this process. The preceding `Evaluating '}} -Instrumentation rules are available for Linux-based apps through a limited availability preview. To configure allow or deny rules for process injection, sign up for preview access. -{{< /callout >}} +Instrumentation rules require Agent v7.73 or later. They let you allow or block Linux processes based on properties such as executable, arguments, working directory, and detected language. See [Define instrumentation rules on Linux][20]. #### Kubernetes instrumentation rules @@ -292,62 +474,81 @@ Instrumentation rules enable injection based on Kubernetes labels and selectors. #### Injection container flagged by security scanners -Security tools may flag the `apm-inject` container because it runs an executable at startup, which can resemble malicious software. +Security tools may flag the `apm-inject` container because it runs an executable at startup, which can resemble malicious software. The container's behavior is expected and safe; the executable configures the environment for auto-instrumentation. -Datadog adheres to security best practices and is working with security vendors to allowlist this container. +Datadog adheres to security best practices and works with security vendors to allowlist this container. ### Environments with strict pod security settings -If pod security rules block the Datadog init container, you may see errors like: +If pod security rules block the Datadog init container when using `init_container` mode, you may see errors like: + ``` Privilege escalation container is not allowed or violates PodSecurity "restricted: latest": allowPrivilegeEscalation is false ``` To resolve this, set one of the following Cluster Agent options: + - `DD_ADMISSION_CONTROLLER_AUTO_INSTRUMENTATION_INIT_SECURITY_CONTEXT` - `admission_controller.auto_instrumentation.init_security_context` -The value should be a JSON string that applies the necessary security context to the Datadog init containers. +The value should be a JSON string that applies the necessary security context to the Datadog init containers. ### Custom instrumentation Custom instrumentation still requires you to import the SDK. Configuration variables like .NET's `DD_TRACE_METHODS` remain available for defining custom spans. -## General troubleshooting +### SDK environment variables do not disable SSI -### SSI continues running after setting DD_TRACE_ENABLED=false +Setting `DD_TRACE_ENABLED=false` does not prevent SSI from loading the SDK. The [injector][8] runs before the SDK evaluates its environment variables, so SDK-level environment variables have no effect on SSI. To disable or remove SSI, see your platform's [SSI setup page][9]. -Setting `DD_TRACE_ENABLED=false` does not prevent SSI from loading the SDK. The [injector][11] runs before the SDK evaluates its environment variables, so SDK-level environment variables have no effect on SSI. To disable or remove SSI, see your platform's [SSI setup page][12]. +## Environment-specific issues -## Environment-specific troubleshooting - -### Host and Docker environments +### Host and Docker #### Host injection does not apply to existing processes -The preload library only injects into newly launched processes. Start a new shell session or log out and log back in to apply instrumentation. +On a Linux host, the preload library only injects into newly launched processes. Restart the application process; if its environment comes from a login session, start a new shell session or log out and back in first. -**Note:** Docker-based injection does not have this limitation. +Docker does not depend on the host login session, but application containers still must be recreated or restarted after SSI is enabled. #### Injection fails on small instance types The preload library allows the analyzer one second to complete its work. On small VM instances running multiple services (for example, `t2.micro`), this time limit may be exceeded. To overcome this issue, use a larger instance size, such as `t2.small`. -#### Errors after manual uninstallation of agent files +#### Stale package registration + +`datadog-installer status` reports what's registered in the installer's package database, not what's present on disk. A package can appear installed while its files are missing. For example, `datadog-installer status` lists `datadog-apm-library-` while `/opt/datadog-packages/datadog-apm-library-/` is empty. When this happens, injection finds no library to load and no traces appear. + +Confirm the on-disk state: + +```shell +ls /opt/datadog-packages/datadog-apm-library-/ +``` + +If the directory is empty or missing, remove the stale registration and reinstall: + +```shell +datadog-installer remove datadog-apm-library- +``` + +Then re-run your platform's SSI installation command. + +#### Errors after manual uninstallation of Agent files + +If you manually delete Agent files, you may see errors like: -If you manually delete agent files, you may see errors like: ``` ERROR: ld.so: object /opt/datadog/apm/inject/launcher.preload.so from /etc/ld.so.preload cannot be preloaded (cannot open shared object file): ignored ``` To properly uninstall SSI, follow the platform-specific instructions: -* [Kubernetes][1] -* [Docker][2] -* [Linux][3] -* [Windows][4] +* [Kubernetes][10] +* [Docker][11] +* [Linux][12] +* [Windows][13] #### Injection not working with rootless Docker @@ -356,12 +557,13 @@ When using rootless Docker, set `docker_socket` in `/etc/datadog-agent/inject/do #### Injection fails with statically linked launchers If a custom launcher is statically linked (common with Go), the preload library might not be invoked. Injection can still succeed if: + - The launcher's command line includes the language name - The launcher runs an intermediary dynamically linked program However, direct process launches from statically linked binaries are not injected. -### Kubernetes environments +### Kubernetes The Datadog Admission Controller must be deployed and configured before application pods are created; it cannot modify existing pods. @@ -402,25 +604,27 @@ The Cluster Agent logs warnings and errors for injection failures, typically fro Use the metric `datadog.cluster_agent.admission_webhooks.library_injection_errors` for further debugging. -##### Language annotation cannot be applied +#### Language annotation cannot be applied During setup, SSI detects the application language of your service and applies a service label in the form `internal.dd.datadoghq.com/service-name.detected_langs`. If the label cannot be applied, injection fails. -Sometimes, labeling errors occur because a service name breaks Kubernetes string limits ([63 characters][6]). For example: +Sometimes, labeling errors occur because a service name breaks Kubernetes string limits ([63 characters][14]). For example: + ``` languagedetection/patcher.go:231 in handleDeploymentEvent) | failed to handle deployment event: annotations: Invalid value: "internal.dd.datadoghq.com/dummy-python-container-long-long-long-long-long-x.detected_langs": name part must be no more than 63 characters ``` -String limit violations are common if service tags are not explicitly set through [Unified Service Tagging][8], in which case default image names are used. +String limit violations are common if service tags are not explicitly set through [Unified Service Tagging][7], in which case default image names are used. -##### Injection appears successful but traces are missing +#### Injection appears successful but traces are missing If logs show no issues but traces are missing, there may be an application-side misconfiguration. Verify that: + - Required annotations and labels are present. -- [Unified Service Tagging][8] is set up correctly. +- [Unified Service Tagging][7] is set up correctly. - Allow/deny lists for instrumentation rules are properly defined. -## Language-specific troubleshooting +## Language-specific issues ### Java @@ -430,7 +634,7 @@ The `JAVA_TOOL_OPTIONS` environment variable has a JVM-enforced limit of 1024 ch To avoid this issue, exclude the affected process from injection. -#### `JAVA_TOOL_OPTIONS` changes program output +#### `JAVA_TOOL_OPTIONS` changes program output When `JAVA_TOOL_OPTIONS` is set, the JVM prints a message to stdout, such as `Picked up JAVA_TOOL_OPTIONS: -Xmx1024m`. If a process reads and depends on this output, it may be affected. @@ -441,6 +645,7 @@ As of version 0.12.2, injection is skipped for `java -version` to avoid interfer By default, Single Step sets the `DD_SERVICE` environment variable, which applies a single service name across all web applications running on the same server (such as Tomcat or WebLogic). As a result, all sites report under the same name. Use one of the following options to enable **split-by-tags** so that each site reports under its own name: + - JVM system property: `-Ddd.trace.split-by-tags=servlet.context` - Environment variable: `DD_TRACE_SPLIT_BY_TAGS=servlet.context` @@ -462,9 +667,9 @@ Versions <=2.7.5 contain a pre-packaged protobuf dependency that can conflict wi #### SSI is applied but no .NET traces reach the Agent -If SSI annotations and init containers are present on the pod but no .NET traces arrive, another profiler may have precedence. Check `CORECLR_PROFILER` on the main container. If the value is not `{846F5F1C-F9AE-4B07-969E-05C26BC060D8}` (the Datadog .NET tracer CLSID), another profiler is loaded instead of Datadog. +If the pod shows SSI mutation but no .NET traces arrive, another profiler may have precedence. Check `CORECLR_PROFILER` on the main container. If the value is not `{846F5F1C-F9AE-4B07-969E-05C26BC060D8}` (the Datadog .NET tracer CLSID), another profiler is loaded instead of Datadog. -Remove the conflicting `CORECLR_*` environment variables (and any `LD_PRELOAD` entries that reference the other profiler) from the source that injected them: another vendor's operator, init container, pod template, or Helm values. Then roll the pods. The [.NET CLR Profiling API allows only one subscriber per process][10]. +Remove the conflicting `CORECLR_*` environment variables (and any `LD_PRELOAD` entries that reference the other profiler) from the source that injected them: another vendor's operator, init container, pod template, or Helm values. Then roll the pods. The [.NET CLR Profiling API allows only one subscriber per process][15]. ## Collect diagnostic information for support @@ -480,7 +685,7 @@ When contacting support about injection issues, collect the following informatio It should be owned by `root` with `644` permissions (`-rw-r--r--`). -1. Enable injector debug logs and collect the output. For instructions, see [Injector debug logs](#injector-debug-logs). +1. Enable injector debug logs and collect the output. For instructions, see [Injector debug logs][17]. 1. Provide an Agent flare. @@ -488,29 +693,35 @@ When contacting support about injection issues, collect the following informatio Collect the following details if troubleshooting injection in a Kubernetes environment: -- The method used to deploy the Cluster Agent (for example, Helm, Datadog Operator, or kubectl commands). +- The method used to deploy the Cluster Agent (for example, Helm or Datadog Operator) and the configured injection mode. - Deployment files for the application pod. - Flares from both the Node Agent and the Cluster Agent, ideally with `DEBUG` mode enabled. - Output of: ``` kubectl describe pod ``` -- Injector debug logs from the application pod (not the Cluster Agent). For instructions, see [Injector debug logs](#injector-debug-logs). +- Injector debug logs from the application pod (not the Cluster Agent). For instructions, see [Injector debug logs][17]. ## Further reading {{< partial name="whats-next/whats-next.html" >}} -[1]: /tracing/trace_collection/automatic_instrumentation/single-step-apm/kubernetes?tab=agentv764recommended#remove-apm-for-all-services-on-the-infrastructure -[2]: /tracing/trace_collection/automatic_instrumentation/single-step-apm/docker#remove-apm-for-all-services-on-the-infrastructure -[3]: /tracing/trace_collection/automatic_instrumentation/single-step-apm/linux#remove-single-step-apm-instrumentation-from-your-agent -[4]: /tracing/trace_collection/automatic_instrumentation/single-step-apm/windows#remove-single-step-apm-instrumentation-from-your-agent -[5]: /containers/guide/sync_container_images/#copy-an-image-to-another-registry-using-crane -[6]: https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/#syntax-and-character-set -[7]: https://datatracker.ietf.org/doc/html/rfc1035 -[8]: /getting_started/tagging/unified_service_tagging/ -[9]: https://app.datadoghq.com/fleet -[10]: /tracing/trace_collection/dd_libraries/dotnet-core/#installation-and-getting-started -[11]: /tracing/guide/injectors/ -[12]: /tracing/trace_collection/automatic_instrumentation/single-step-apm/#instrument-sdks-across-applications -[13]: /tracing/troubleshooting/tracer_debug_logs/ +[1]: /help/ +[2]: /tracing/trace_collection/single-step-apm/ +[3]: /tracing/trace_collection/single-step-apm/compatibility/ +[5]: https://app.datadoghq.com/fleet +[6]: /tracing/troubleshooting/tracer_debug_logs/ +[7]: /getting_started/tagging/unified_service_tagging/ +[8]: /tracing/guide/injectors/ +[9]: /tracing/trace_collection/single-step-apm/#instrument-sdks-across-applications +[10]: /tracing/trace_collection/single-step-apm/kubernetes/#remove-instrumentation-for-all-services +[11]: /tracing/trace_collection/single-step-apm/docker#remove-apm-for-all-services-on-the-infrastructure +[12]: /tracing/trace_collection/single-step-apm/linux#remove-single-step-apm-instrumentation-from-your-agent +[13]: /tracing/trace_collection/single-step-apm/windows#remove-single-step-apm-instrumentation-from-your-agent +[14]: https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/#syntax-and-character-set +[15]: /tracing/trace_collection/dd_libraries/dotnet-core/#installation-and-getting-started +[16]: #injection-appears-successful-but-traces-are-missing +[17]: #injector-debug-logs +[18]: /tracing/guide/injectors/#per-runtime-instrumentation +[19]: #troubleshoot-injection-in-datadog-fleet-automation +[20]: /tracing/trace_collection/single-step-apm/linux/#define-instrumentation-rules diff --git a/content/en/tracing/trace_collection/single-step-apm/windows.md b/content/en/tracing/trace_collection/single-step-apm/windows.md index 5cb4fbb0485..97e1a0062a0 100644 --- a/content/en/tracing/trace_collection/single-step-apm/windows.md +++ b/content/en/tracing/trace_collection/single-step-apm/windows.md @@ -1,5 +1,6 @@ --- title: Single Step APM Instrumentation on Windows +description: "Enable Datadog Single Step Instrumentation for .NET applications in IIS or Java and .NET applications across a Windows host." code_lang: windows type: multi-code-lang code_lang_weight: 30 @@ -13,40 +14,47 @@ further_reading: ## Overview -With Single Step Instrumentation (SSI), you can enable APM for your Java and .NET applications on Windows VMs using a single Datadog Agent installation command. +With Single Step Instrumentation (SSI), you enable APM for your Windows applications by adding options to a single Datadog Agent installation command. -## Enable APM on Windows +- Instrumenting .NET applications on IIS is generally available (Agent v7.67.1+, .NET SDK v3.19.0+). +- Host-wide instrumentation of Java and .NET applications is in Preview. -
Before proceeding, confirm that your environment is compatible by reviewing the SSI compatibility guide.
+Before you begin, confirm that your environment meets the [SSI compatibility requirements][8]. -### New Agent installation +## Enable APM on Windows -If you don't yet have a Datadog Agent installed, follow these steps to install the Agent and enable SSI simultaneously. +### New Agent installation -You can enable APM on Windows in the following ways: -* Instrument only .NET applications on IIS -* Instrument all Java and .NET applications across your entire Windows host +If you don't yet have a Datadog Agent installed, install the Agent and enable SSI in one step. {{< tabs >}} -{{% tab "IIS" %}} +{{% tab "IIS (.NET)" %}} -To instrument only .NET applications running on IIS: +To instrument .NET applications running on IIS, run the following command in an elevated PowerShell session. Replace `` with your [Datadog API key][1] and `` with your [Datadog site][2] ({{< region-param key="dd_site" code="true" >}}): -1. In Datadog, go to [Install the Datadog Agent on Windows][1]. -1. In the {{< ui >}}Customize your observability coverage{{< /ui >}} section, toggle {{< ui >}}Application Performance Monitoring (APM){{< /ui >}}. -1. (Optional) Set your SDK version: - - By default, Single Step Instrumentation installs the latest supported version of the Datadog .NET SDK. If you need to pin a specific version: +```powershell +$p = Start-Process -Wait -PassThru msiexec -ArgumentList '/qn /norestart /i "https://windows-agent.datadoghq.com/datadog-agent-7-latest.amd64.msi" /log C:\Windows\SystemTemp\install-datadog.log APIKEY="" SITE="" DD_APM_INSTRUMENTATION_ENABLED="iis"' +if ($p.ExitCode -ne 0) { + Write-Host "msiexec failed with exit code $($p.ExitCode) please check the logs at C:\Windows\SystemTemp\install-datadog.log" -ForegroundColor Red +} +``` + +By default, SSI installs the latest supported version of the Datadog .NET SDK. To pin a specific version, add the `DD_APM_INSTRUMENTATION_LIBRARIES` property to the argument list, for example `DD_APM_INSTRUMENTATION_LIBRARIES="dotnet:3"`. - 1. Under {{< ui >}}Instrumentation Configuration{{< /ui >}}, select {{< ui >}}Customize Library Versions{{< /ui >}}. - 1. Under .NET, choose the version you want to use. - -1. Copy and run the provided MSI install command on your Windows host. -1. Restart the IIS applications you want instrumented. (You do not need to restart the entire IIS server.) +After installation, the Agent loads the Datadog .NET SDK into supported application processes to enable distributed tracing. -After installation, the Agent automatically loads the Datadog .NET SDK into supported application processes to enable distributed tracing. +{{< collapse-content title="Alternate: install with the in-app wizard" level="h4" >}} -[1]: https://app.datadoghq.com/fleet/install-agent/latest?platform=windows +1. In Datadog, go to [Install the Datadog Agent on Windows][3]. +1. In the {{< ui >}}Customize your observability coverage{{< /ui >}} section, toggle {{< ui >}}Application Performance Monitoring (APM){{< /ui >}}. +1. (Optional) Under {{< ui >}}Instrumentation Configuration{{< /ui >}}, select {{< ui >}}Customize Library Versions{{< /ui >}}, then choose the .NET version to pin. +1. Copy and run the generated MSI installation command on your Windows host. + +{{< /collapse-content >}} + +[1]: https://app.datadoghq.com/organization-settings/api-keys +[2]: /getting_started/site/ +[3]: https://app.datadoghq.com/fleet/install-agent/latest?platform=windows {{% /tab %}} @@ -57,25 +65,31 @@ After installation, the Agent automatically loads the Datadog .NET SDK into supp Host-wide instrumentation for Windows is in Preview and limited to Preview participants. The installation and configuration options described in this tab appear in Datadog only after you are enrolled. Request access to join the Preview. -To instrument Java and .NET applications across your entire Windows host: +After you are enrolled, instrument Java and .NET applications across your entire Windows host by running the following command in an elevated PowerShell session. Replace `` with your [Datadog API key][1] and `` with your [Datadog site][2] ({{< region-param key="dd_site" code="true" >}}): -1. In Datadog, go to [Install the Datadog Agent on Windows][1]. -1. In the {{< ui >}}Customize your observability coverage{{< /ui >}} section, toggle {{< ui >}}Application Performance Monitoring (APM){{< /ui >}}. -1. (Optional) Set your SDK version: - - By default, Single Step Instrumentation installs the latest supported version of the Datadog .NET and Java SDK. If you need to pin a specific version: +```powershell +$p = Start-Process -Wait -PassThru msiexec -ArgumentList '/qn /norestart /i "https://windows-agent.datadoghq.com/datadog-agent-7-latest.amd64.msi" /log C:\Windows\SystemTemp\install-datadog.log APIKEY="" SITE="" DD_APM_INSTRUMENTATION_ENABLED="host"' +if ($p.ExitCode -ne 0) { + Write-Host "msiexec failed with exit code $($p.ExitCode) please check the logs at C:\Windows\SystemTemp\install-datadog.log" -ForegroundColor Red +} +``` - 1. Under {{< ui >}}Instrumentation Configuration{{< /ui >}}, select {{< ui >}}Customize Library Versions{{< /ui >}}. - 1. Under .NET, choose the version you want to use. +By default, SSI installs the latest supported versions of the Datadog .NET and Java SDKs. To pin specific versions, add the `DD_APM_INSTRUMENTATION_LIBRARIES` property, for example `DD_APM_INSTRUMENTATION_LIBRARIES="dotnet:3,java:1"`. -1. Copy and run the provided MSI install command on your Windows host. -1. Configure instrumentation rules. +Host-wide SSI instruments all Java applications on the host and all .NET applications running in IIS. To instrument .NET applications running outside of IIS, or for granular control over which processes are instrumented, [define instrumentation rules][9]. - Host-wide SSI automatically instruments all Java applications on the host and all .NET applications running in IIS. To instrument .NET applications running outside of IIS, you must [define an instrumentation rule](#define-instrumentation-rules) that allows them. You can also use instrumentation rules for granular control over which Java applications on the host or .NET applications in IIS are instrumented. +{{< collapse-content title="Alternate: install with the in-app wizard" level="h4" >}} -1. Restart the services you want instrumented. +1. In Datadog, go to [Install the Datadog Agent on Windows][3]. +1. In the {{< ui >}}Customize your observability coverage{{< /ui >}} section, toggle {{< ui >}}Application Performance Monitoring (APM){{< /ui >}}. +1. (Optional) Under {{< ui >}}Instrumentation Configuration{{< /ui >}}, select {{< ui >}}Customize Library Versions{{< /ui >}}, then choose the SDK versions to pin. +1. Copy and run the generated MSI installation command on your Windows host. -[1]: https://app.datadoghq.com/fleet/install-agent/latest?platform=windows +{{< /collapse-content >}} + +[1]: https://app.datadoghq.com/organization-settings/api-keys +[2]: /getting_started/site/ +[3]: https://app.datadoghq.com/fleet/install-agent/latest?platform=windows {{% /tab %}} {{< /tabs >}} @@ -102,13 +116,30 @@ If you already have a Datadog Agent installed, use Fleet Automation to enable SS 1. Click {{< ui >}}Next{{< /ui >}}. 1. Review your configuration and click {{< ui >}}Deploy Configuration{{< /ui >}}. -1. Configure instrumentation rules. - Host-wide SSI automatically instruments all Java applications on the host and all .NET applications running in IIS. To instrument .NET applications running outside of IIS, you must [define an instrumentation rule](#define-instrumentation-rules) that allows them. You can also use instrumentation rules for granular control over which Java applications on the host or .NET applications in IIS are instrumented. +For host-wide instrumentation, you can [define instrumentation rules][9] to instrument .NET applications running outside of IIS or to control which processes are instrumented. + +## Verify your first trace + +After you enable SSI, restart your application, generate traffic, and confirm that traces reach Datadog. + +For .NET on IIS: + +1. Recycle the IIS application pool for the app you want to instrument. A full server restart is not required: + + ```powershell + Restart-WebAppPool -Name "" + ``` + +1. Confirm that the Datadog profiler is loaded. In the instrumented worker process, `CORECLR_PROFILER` must equal `{846F5F1C-F9AE-4B07-969E-05C26BC060D8}`. If another profiler takes precedence, no Datadog traces flow. Remove any conflicting `CORECLR_*` environment variables so the Datadog profiler is used. +1. Generate traffic to your application. +1. In Datadog, confirm that your service appears in the [APM Services page][7]. + +If traces don't appear, see the [SSI troubleshooting guide][4]. ## Configure Unified Service Tags -Unified Service Tags (USTs) apply consistent tags across traces, metrics, and logs, making it easier to navigate and correlate your observability data. Learn how to [set USTs for Windows services][2]. +Unified Service Tags (USTs) apply consistent tags across traces, metrics, and logs, so you can navigate and correlate your observability data. Learn how to [set USTs for Windows services][2]. ## Enable SDK-dependent products and features @@ -118,16 +149,30 @@ After SSI loads the Datadog SDK into your applications and enables distributed t To enable products, [set environment variables][3] in your application configuration. -## Advanced options +## Remove Single Step APM instrumentation from your Agent + +### IIS-only instrumentation + +To remove the SSI-managed .NET SDK package, run the following command in an elevated PowerShell session, then recycle the affected IIS application pools: + +```powershell +&"C:\Program Files\Datadog\Datadog Agent\bin\datadog-installer.exe" remove datadog-apm-library-dotnet +``` -### Define instrumentation rules +### Host-wide instrumentation (Preview) + +Removing only the .NET SDK package does not disable the Windows host injector or remove the Java SDK. To remove host-wide instrumentation, roll back the SSI configuration through Fleet Automation or follow the offboarding instructions provided with the Preview, then restart affected applications. Contact the Preview team or [Datadog Support][10] if the rollback control is not available in your account. + +## Advanced configuration + +{{< collapse-content title="Define instrumentation rules" level="h3" id="define-instrumentation-rules" >}} {{< site-region region="gov" >}}
Instrumentation rules are not supported for your selected Datadog site ({{< region-param key="dd_site_name" >}}).
{{< /site-region >}} {{< site-region region="us,us3,us5,eu,ap1,ap2,uk1" >}} -
Instrumentation rules (available for Agent v7.73+) apply only to host-wide instrumentation. They are not supported for IIS-only installation.
+
Instrumentation rules require Agent v7.73+ and apply only to host-wide instrumentation. They are not supported for IIS-only installation.
{{< /site-region >}} Instrumentation rules let you control which processes are automatically instrumented by SSI on Windows hosts. Rules are required to instrument .NET applications running outside of IIS. They are also useful for granular control over which Java applications on the host or .NET applications in IIS are instrumented. @@ -139,7 +184,7 @@ To configure instrumentation rules: 1. Define instrumentation rules: 1. Click {{< ui >}}Add New Rule{{< /ui >}}, then choose {{< ui >}}Allow Rule{{< /ui >}} or {{< ui >}}Block Rule{{< /ui >}} to specify whether matching processes should be instrumented. 1. Name your rule. - 1. Add one or more conditions. See [Define rule conditions](#define-rule-conditions) to learn more. + 1. Add one or more conditions. See [Define rule conditions][11] to learn more. {{< img src="tracing/trace_collection/define_instrumentation_rule.png" alt="The instrumentation rules UI, showing configuration options for defining a rule" style="width:100%;" >}} @@ -186,19 +231,13 @@ Instrument all processes by default. Add block rules to exclude services that wo {{< collapse-content title="Example 2: Instrument only specific IIS applications" level="h5" >}} -Block all instrumentation by default. Add allow rules to opt specific IIS applications into APM. Because all IIS workers share the w3wp.exe executable, use {{< ui >}}IIS Application Pool{{< /ui >}} to identify target applications. This approach is useful for gradual rollouts. +Block all instrumentation by default. Add allow rules to opt specific IIS applications into APM. Because all IIS workers share the `w3wp.exe` executable, use {{< ui >}}IIS Application Pool{{< /ui >}} to identify target applications. This approach is useful for gradual rollouts. {{< img src="tracing/trace_collection/instrumentation-rules-example-2.png" alt="Two allow instrumentation rules targeting specific IIS application pools by name, with a default of block instrumentation" style="width:100%;" >}} {{< /collapse-content >}} -## Remove Single Step APM instrumentation from your Agent - -To disable SSI for .NET on your host, run: - -```shell -&"C:\Program Files\Datadog\Datadog Agent\bin\datadog-installer.exe" remove datadog-apm-library-dotnet -``` +{{< /collapse-content >}} ## Troubleshooting @@ -208,9 +247,13 @@ If you encounter problems enabling APM with SSI, see the [SSI troubleshooting gu {{< partial name="whats-next/whats-next.html" >}} -[1]: https://app.datadoghq.com/fleet/install-agent/latest?platform=windows [2]: /integrations/windows-service/#tags [3]: /tracing/trace_collection/library_config/ -[4]: /tracing/trace_collection/automatic_instrumentation/single-step-apm/troubleshooting +[4]: /tracing/trace_collection/single-step-apm/troubleshooting/ [5]: https://app.datadoghq.com/apm/service-setup/workload-selection [6]: https://app.datadoghq.com/fleet/agent-management +[7]: https://app.datadoghq.com/apm/services +[8]: /tracing/trace_collection/single-step-apm/compatibility/ +[9]: #define-instrumentation-rules +[10]: /help/ +[11]: #define-rule-conditions