From dd6a3ac1197b4338c4e6904703d7ef14f0006771 Mon Sep 17 00:00:00 2001 From: Brett Blue Date: Wed, 29 Apr 2026 14:00:43 -0600 Subject: [PATCH 01/13] Restructure APM onboarding IA and K8s SSI page for AI readability Rename Application Instrumentation to Set Up APM in the nav. Rework the entry page as a task-oriented routing page. Rewrite the Kubernetes SSI page with a clear onboarding flow, verification steps, agent-only diagnostic content, and tighter advanced configuration sections. Add a symptom-first triage section to SSI troubleshooting. --- config/_default/menus/main.en.yaml | 2 +- content/en/tracing/trace_collection/_index.md | 147 +-- .../single-step-apm/kubernetes.md | 872 ++++++++---------- .../single-step-apm/troubleshooting.md | 52 ++ 4 files changed, 462 insertions(+), 611 deletions(-) diff --git a/config/_default/menus/main.en.yaml b/config/_default/menus/main.en.yaml index 0b52770f73a..bc2d9fd168a 100644 --- a/config/_default/menus/main.en.yaml +++ b/config/_default/menus/main.en.yaml @@ -4054,7 +4054,7 @@ menu: parent: tracing identifier: tracing_glossary weight: 1 - - name: Application Instrumentation + - name: Set Up APM url: tracing/trace_collection/ parent: tracing identifier: tracing_trace_collection diff --git a/content/en/tracing/trace_collection/_index.md b/content/en/tracing/trace_collection/_index.md index 1ed02680a2f..237743d6a2f 100644 --- a/content/en/tracing/trace_collection/_index.md +++ b/content/en/tracing/trace_collection/_index.md @@ -1,5 +1,5 @@ --- -title: Application Instrumentation +title: Set Up APM description: "Get Started with Datadog APM" aliases: - /tracing/setup @@ -31,148 +31,55 @@ algolia: --- ## Overview -Application {{< tooltip glossary="instrumentation" >}} with Datadog APM involves: -1. **SDK setup**: Adding a Datadog SDK to your application. -2. **Span creation**: Capturing observability data as {{< tooltip glossary="span" >}}s. +Set up Datadog APM to collect traces from your applications. [Single Step Instrumentation][1] (SSI) is the recommended approach for most users. It automatically installs and configures Datadog SDKs with no code changes required. - Spans are automatically generated by default as soon as the SDK is loaded. This is known as **auto-instrumentation** and provides sufficient visibility for most users. If you need more control, you can optionally add custom spans. - -**Note**: These steps assume you have a [Datadog Agent][5] installed and configured to receive traces. - -{{< img src="tracing/visualization/troubleshooting_pipeline.png" alt="The APM pipeline">}} - -## Getting started +## Get APM running
Prefer vendor-neutral instrumentation? See the OpenTelemetry documentation for using OpenTelemetry with Datadog.
-### Single step instrumentation (recommended) - -[Single Step Instrumentation][1] (SSI) automatically installs and configures Datadog SDKs with a single command. Auto-instrumentation then immediately begins capturing traces from your supported frameworks and libraries, with no code changes required. +SSI automatically installs and loads Datadog SDKs into your application processes at runtime with a single command. Pick your platform to get started: {{< whatsnext desc=" " >}} - {{< nextlink href="/tracing/trace_collection/single-step-apm/" >}}Get started with Single Step Instrumentation{{< /nextlink >}} + {{< nextlink href="/tracing/trace_collection/single-step-apm/kubernetes" >}}Kubernetes{{< /nextlink >}} + {{< nextlink href="/tracing/trace_collection/single-step-apm/linux" >}}Linux{{< /nextlink >}} + {{< nextlink href="/tracing/trace_collection/single-step-apm/docker" >}}Docker{{< /nextlink >}} + {{< nextlink href="/tracing/trace_collection/single-step-apm/windows" >}}Windows IIS{{< /nextlink >}} {{< /whatsnext >}} -### Manual setup and custom spans - -As your observability needs grow, you can add more control and customization: +## Need more control? -**For full SDK configuration control:** Use [manually managed Datadog SDKs][2] if you need granular control over SDK behavior and configuration. +If you need full control over SDK configuration and installation, use [manually managed Datadog SDKs][2]. This approach requires adding the SDK to your application code or build process. {{< whatsnext desc=" " >}} - {{< nextlink href="/tracing/trace_collection/dd_libraries/" >}}Use manually managed Datadog SDKs{{< /nextlink >}} + {{< nextlink href="/tracing/trace_collection/dd_libraries/" >}}Set up manually managed SDKs{{< /nextlink >}} {{< /whatsnext >}} -**For custom spans without code changes:** Use [Dynamic Instrumentation][4] to create custom spans from the Datadog UI without redeploying your application. +## Add custom spans -{{< whatsnext desc=" " >}} - {{< nextlink href="/tracing/trace_collection/dynamic_instrumentation/" >}}Add custom spans with Dynamic Instrumentation{{< /nextlink >}} -{{< /whatsnext >}} +After you have APM running, you can add custom spans for visibility into application-specific code paths: -**For custom spans in code:** Add [code-based custom instrumentation][3] to instrument custom business logic or add application-specific metadata to spans. +- **Without code changes**: Use [Dynamic Instrumentation][4] to create custom spans from the Datadog UI without redeploying your application. +- **In code**: Add [code-based custom instrumentation][3] to instrument custom business logic or add application-specific metadata to spans. {{< whatsnext desc=" " >}} + {{< nextlink href="/tracing/trace_collection/dynamic_instrumentation/" >}}Add custom spans with Dynamic Instrumentation{{< /nextlink >}} {{< nextlink href="/tracing/trace_collection/custom_instrumentation/" >}}Add custom spans with code-based instrumentation{{< /nextlink >}} {{< /whatsnext >}} -These options can be combined. For example, you can start with Single Step Instrumentation and add code-based custom instrumentation for specific spans, or use manually managed SDKs with Dynamic Instrumentation for no-deploy span additions. - -## Detailed comparison - -### SDK setup - -Single Step Instrumentation is the recommended starting point for most users. If you need more control over SDK configuration, you can use manually managed SDKs instead: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Single Step Instrumentation (recommended)Manually managed SDKs
How it worksDatadog automatically installs and loads SDKs into your application processes, at runtime, with a single command.You install and configure SDKs directly in your application code or build process.
Code changes?NoYes
Setup complexityLow - minimal configuration neededMedium - requires environment and build configuration
Configuration controlStandard defaults with optional overridesFull control through environment variables and code
When to useStart here for fast, consistent instrumentation across services without code changes.Progress to this when you need granular control over SDK behavior and configuration.
- -### Span customization - -Auto-instrumentation automatically creates spans for supported frameworks and libraries, providing essential observability with no additional work. When you need visibility into custom code paths or want to enrich traces with application-specific data, you can add custom spans using either Dynamic Instrumentation or code-based custom instrumentation: - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Dynamic InstrumentationCode-based custom instrumentation
How it worksConfigure instrumentation rules in the Datadog UI; rules are applied at runtime.Add explicit tracing API calls in your application code.
Code changes?NoYes
Deployment requiredNoYes (to add or modify spans)
When to useAdd custom spans without code changes or redeployments.Progress to this when you need complex instrumentation logic or want spans permanently defined in code.
- -## APM setup tutorials - -The following tutorials guide you through setting up distributed tracing for a sample application on various infrastructure scenarios, with both automatic and custom instrumentation: - -{{< whatsnext desc="Choose your language and environment:" >}} - {{< nextlink href="tracing/guide/tutorial-enable-python-host" >}} Enabling Tracing on a Python Application on the Same Host as Datadog Agent{{< /nextlink >}} - {{< nextlink href="tracing/guide/tutorial-enable-python-containers" >}} Enabling Tracing on a Python Application and Datadog Agent in Containers{{< /nextlink >}} - {{< nextlink href="tracing/guide/tutorial-enable-python-container-agent-host" >}} Enabling Tracing for a Python Application in a Container and an Agent on a Host{{< /nextlink >}} - {{< nextlink href="tracing/guide/tutorial-enable-java-host" >}} Enabling Tracing on a Java Application on the Same Host as Datadog Agent{{< /nextlink >}} - {{< nextlink href="tracing/guide/tutorial-enable-java-containers" >}} Enabling Tracing on a Java Application and Datadog Agent in Containers{{< /nextlink >}} - {{< nextlink href="tracing/guide/tutorial-enable-java-container-agent-host" >}} Enabling Tracing for a Java Application in a Container and an Agent on a Host{{< /nextlink >}} - {{< nextlink href="tracing/guide/tutorial-enable-java-gke" >}} Enabling Tracing for a Java Application on GKE{{< /nextlink >}} - {{< nextlink href="tracing/guide/tutorial-enable-java-aws-eks" >}} Enabling Tracing for a Java Application on Amazon EKS{{< /nextlink >}} - {{< nextlink href="tracing/guide/tutorial-enable-java-aws-ecs-ec2" >}} Enabling Tracing for a Java Application in Amazon ECS with EC2{{< /nextlink >}} - {{< nextlink href="tracing/guide/tutorial-enable-java-aws-ecs-fargate" >}} Enabling Tracing for a Java Application in Amazon ECS with Fargate{{< /nextlink >}} - {{< nextlink href="tracing/guide/tutorial-enable-java-admission-controller" >}} Enabling Tracing for a Java Application with the Admission Controller{{< /nextlink >}} - {{< nextlink href="tracing/guide/tutorial-enable-go-host" >}} Enabling Tracing on a Go Application on the Same Host as Datadog Agent{{< /nextlink >}} - {{< nextlink href="tracing/guide/tutorial-enable-go-containers" >}} Enabling Tracing on a Go Application and Datadog Agent in Containers{{< /nextlink >}} - {{< nextlink href="tracing/guide/tutorial-enable-go-aws-ecs-ec2" >}} Enabling Tracing for a Go Application in Amazon ECS with EC2{{< /nextlink >}} - {{< nextlink href="tracing/guide/tutorial-enable-go-aws-ecs-fargate" >}} Enabling Tracing for a Go Application in Amazon ECS with Fargate{{< /nextlink >}} +{{< agent-only >}} +If a user asks to set up APM, tracing, or distributed tracing, start with Single Step Instrumentation (SSI) unless they specifically request manual SDK control. -{{< /whatsnext >}} +Ask what platform they're running on: +- Kubernetes → /tracing/trace_collection/single-step-apm/kubernetes +- Linux host or VM → /tracing/trace_collection/single-step-apm/linux +- Docker → /tracing/trace_collection/single-step-apm/docker +- Windows IIS → /tracing/trace_collection/single-step-apm/windows + +If their language is Go, C++, or Rust, SSI is not available — direct them to manually managed SDKs at /tracing/trace_collection/dd_libraries/. +{{< /agent-only >}} ## Further reading @@ -182,5 +89,3 @@ The following tutorials guide you through setting up distributed tracing for a s [2]: /tracing/trace_collection/dd_libraries/ [3]: /tracing/trace_collection/custom_instrumentation/ [4]: /tracing/trace_collection/dynamic_instrumentation/ -[5]: /agent/ - diff --git a/content/en/tracing/trace_collection/single-step-apm/kubernetes.md b/content/en/tracing/trace_collection/single-step-apm/kubernetes.md index 82a16df0d83..c7ce234f7e4 100644 --- a/content/en/tracing/trace_collection/single-step-apm/kubernetes.md +++ b/content/en/tracing/trace_collection/single-step-apm/kubernetes.md @@ -1,5 +1,6 @@ --- title: Single Step APM Instrumentation on Kubernetes +description: "Enable Single Step Instrumentation (SSI) on Kubernetes to automatically instrument applications with Datadog APM. Covers prerequisites, setup with Helm or the Datadog Operator, verification, Unified Service Tags, workload targeting, and troubleshooting." code_lang: kubernetes type: multi-code-lang code_lang_weight: 20 @@ -12,97 +13,263 @@ further_reading: - link: /tracing/guide/init_resource_calc/ tag: Documentation text: Learn about init container resource usage - - link: /tracing/guide/local_sdk_injection + - link: /tracing/trace_collection/single-step-apm/troubleshooting tag: Documentation - text: Instrument your applications using local SDK injection - - link: "https://learn.datadoghq.com/courses/configuring-ssi-k8s" - tag: "Learning Center" - text: "Configuring Single Step Instrumentation on Kubernetes" - + text: Troubleshoot Single Step APM + - link: /tracing/guide/injectors/ + tag: Documentation + text: Understanding injector behavior with Single Step Instrumentation --- ## Overview -In a Kubernetes environment, use Single Step Instrumentation (SSI) for APM to install the Datadog Agent and [instrument][3] your applications with the Datadog SDKs in one step. +Single Step Instrumentation (SSI) installs the Datadog Agent and instruments your Kubernetes applications in one step. SSI loads the Datadog SDK into your application processes at runtime, with no code changes or image rebuilds required. After you enable SSI, all supported applications in your cluster automatically begin sending traces to Datadog. + +## Prerequisites + +- Kubernetes v1.20+ +- [Helm][1] for deploying the Datadog Operator +- [kubectl][2] for installing the Datadog Agent +- A supported language runtime per the [SSI compatibility guide][36] +- **No existing tracer dependencies.** SSI silently disables itself if it detects `ddtrace`, `dd-trace`, an OpenTelemetry SDK, or `-javaagent` in your application. Before enabling SSI, check your dependency manifests: + ``` + grep -rn "ddtrace\|dd-trace\|opentelemetry" requirements.txt package.json Gemfile go.mod pom.xml build.gradle 2>/dev/null + ``` + Remove any matches and rebuild your application image before proceeding. +- **Node.js: CommonJS only.** SSI does not support ECMAScript Modules (ESM). If your application uses `import` syntax or sets `"type": "module"` in `package.json`, use [manually managed SDKs][44] instead. +- **Alpine and musl-based images are supported.** Kubernetes SSI injects through the `LD_PRELOAD` environment variable, not `/etc/ld.so.preload`, so musl libc images work without issues. + +{{< agent-only >}} +Resolve these variables before starting: + +| Variable | How to resolve | +|---|---| +| CLUSTER_NAME | `kubectl config current-context` or `spec.global.clusterName` in datadog-agent.yaml | +| AGENT_NAMESPACE | Namespace where the Datadog Agent is installed (default: `datadog`) | +| APP_NAMESPACE | Namespace of the user's application — ask the user | +| DEPLOYMENT_NAME | `metadata.name` in the application Deployment — ask the user or check their repo | +| APP_LABEL | `spec.selector.matchLabels.app` in the Deployment | +| SERVICE_NAME | `tags.datadoghq.com/service` label on the Deployment — ask the user | +| ENV | `tags.datadoghq.com/env` label on the Deployment — ask the user | + +Pre-flight checks: +- `kubectl config current-context` returns the target cluster +- `helm version` confirms Helm is installed +- Check for existing Datadog deployment: `helm list -A | grep datadog` +- The user has a Datadog API key (do not ask them to share it in chat) +- Application language is one of: Java, Python, Ruby, Node.js, .NET, PHP +- Runtime version is within the SSI supported range per the compatibility guide +- For Node.js: confirm the app uses CommonJS, not ESM (`"type": "module"` in package.json means ESM) + +Check for existing instrumentation that silently disables SSI. Check both dependency manifests AND source files: +- Dependency manifests: `grep -rn "ddtrace\|dd-trace\|opentelemetry\|dd-java-agent\|javaagent" requirements.txt package.json Gemfile go.mod pom.xml build.gradle` +- Python: `grep -rn "import ddtrace\|from ddtrace\|ddtrace.patch_all" *.py` +- Node.js: `grep -rn "require.*dd-trace\|require.*ddtrace\|DD.init" *.js` +- Java: look for `-javaagent` in Dockerfile CMD/ENTRYPOINT or `JAVA_TOOL_OPTIONS` +- .NET: `grep -rn "Tracer.Instance\|DD.Trace" *.cs` +- Ruby: `grep -rn "require.*ddtrace\|Datadog.configure" *.rb` +- PHP: `grep -rn "DDTrace" *.php` + +Do NOT add `ddtrace-run` or `ddtrace` imports. SSI handles instrumentation automatically. Manual instrumentation causes SSI to silently disable itself. + +Always configure SSI through the DatadogAgent manifest (Helm values or Operator CR). Never use `kubectl patch`. Always use `kubectl apply -f` with a YAML file. +{{< /agent-only >}} + +## Enable SSI + +
SSI does not instrument applications in the namespace where the Datadog Agent is installed. Install the Agent in a separate namespace.
+ +These steps enable SSI across your entire cluster. To instrument specific namespaces or pods, see [Target specific workloads](#target-specific-workloads). -## Requirements +{{< tabs >}} +{{% tab "In-app wizard" %}} -- Kubernetes v1.20+. -- [`Helm`][1] for deploying the Datadog Operator. -- [`Kubectl` CLI][2] for installing the Datadog Agent. -- Confirmed environment compatibility per the [Single Step Instrumentation compatibility guide][36]. +The Datadog app generates a configuration file with SSI enabled: +1. Go to the [Install the Datadog Agent on Kubernetes][11] page. +1. Choose your installation method, select an API key, and set up the Operator or Helm repository. +1. Under **Configure `datadog-agent.yaml`**, go to **Additional configuration** > **Application Observability** and turn on **APM Instrumentation**. -## Enable APM on your applications + {{< img src="tracing/trace_collection/k8s-apm-instrumentation-toggle.jpg" alt="APM Instrumentation toggle in the Kubernetes Agent installation wizard" style="width:100%;" >}} -
Single Step Instrumentation does not instrument applications in the namespace where the Datadog Agent is installed. Install the Agent in a separate namespace where you do not run your applications.
+1. Deploy the Agent with the generated configuration file. +1. Restart your application pods: + ```shell + kubectl rollout restart deployment/ -n + ``` -Follow these steps to enable Single Step Instrumentation across your entire cluster. This automatically sends traces from all applications written in supported languages. +[11]: https://app.datadoghq.com/fleet/install-agent/latest?platform=kubernetes -**Note:** To instrument only specific namespaces or pods, see workload targeting in [Advanced options](#advanced-options). +{{% /tab %}} +{{% tab "Helm" %}} -1. In Datadog, go to the [Install the Datadog Agent on Kubernetes][11] page. -1. Follow the on-screen instructions to choose your installation method, select an API key, and set up the Operator or Helm repository. -1. In the **Configure `datadog-agent.yaml`** section, go to **Additional configuration** > **Application Observability**, and turn on **APM Instrumentation**. +Add the following to your `datadog-values.yaml`: - {{< img src="tracing/trace_collection/k8s-apm-instrumentation-toggle.jpg" alt="The configuration block for installing the Datadog Agent on Kubernetes through the Datadog app" style="width:100%;" >}} +```yaml +datadog: + apm: + instrumentation: + enabled: true +``` -1. Deploy the Agent using the generated configuration file. -1. Restart your applications. +Deploy the Agent: -
SSI adds a small amount of startup time to instrumented applications. If this overhead is not acceptable for your use case, contact Datadog Support.
+```shell +helm upgrade datadog-agent -f datadog-values.yaml datadog/datadog +``` -## Configure Unified Service Tags +Restart your application pods: -Unified Service Tags (USTs) apply consistent tags across traces, metrics, and logs, making it easier to navigate and correlate your observability data. You can configure USTs through automatic label extraction (recommended), through explicit configuration with `ddTraceConfigs`, or in deployment manifests. +```shell +kubectl rollout restart deployment/ -n +``` -
-If you are using Remote Configuration, automatic label extraction is not compatible. You must configure USTs explicitly using ddTraceConfigs. -
+{{% /tab %}} +{{% tab "Datadog Operator" %}} -### (Recommended) Configure USTs through automatic label extraction +Add the following to your `datadog-agent.yaml`: -With SSI, you can automatically extract UST values from pod labels and metadata without modifying individual deployments. To do this, configure `kubernetesResourcesLabelsAsTags` to map your existing Kubernetes labels to Datadog service tags. +```yaml +features: + apm: + instrumentation: + enabled: true +``` -**Note:** This method is not compatible with Remote Configuration. If you're using Remote Configuration, see [Configure USTs explicitly with ddTraceConfigs](#configure-usts-explicitly-with-ddtraceconfigs). +Apply the manifest: + +```shell +kubectl apply -f datadog-agent.yaml +``` + +Restart your application pods: + +```shell +kubectl rollout restart deployment/ -n +``` + +{{% /tab %}} +{{< /tabs >}} -#### Prerequisites +
SSI adds a small amount of startup time to instrumented applications. If this overhead is not acceptable, contact Datadog Support.
-| Component | Minimum version | -|-----------|------------------| -| `datadog-agent` | 7.69 | -| `datadog-operator` | 1.16.0 | -| `datadog-helm-chart` | 3.120.0 | +
If your cluster enforces PodSecurity restricted policies, the Datadog init container may be blocked with: allowPrivilegeEscalation is false or violates PodSecurity "restricted:latest". See the SSI troubleshooting guide for the required security context configuration.
-#### Configuration +{{< agent-only >}} +Use the Helm or Datadog Operator tab above. Add `instrumentation.enabled: true` to the user's existing manifest. Do not create a separate manifest. -Replace `app.kubernetes.io/name` in the following example with any label that contains your service name (for example, `service.kubernetes.io/name` or `component`). You can configure multiple labels this way. +If the user wants to pin SDK versions, recommend these defaults: +`java: "1"`, `python: "2"`, `js: "5"`, `dotnet: "3"`, `ruby: "2"`, `php: "1"` + +After applying the config, confirm with the user before restarting their pods. After restart, wait for pods to be ready: +`kubectl wait --for=condition=Ready pod -l app= -n --timeout=120s` + +If `enabledNamespaces` or `disabledNamespaces` is set, SSI only instruments matching namespaces. You cannot set both. See the Target specific workloads section for workload targeting with `targets` and `podSelector`. +{{< /agent-only >}} + +## Verify SSI is working + +After restarting your application pods, wait two to three minutes for traces to arrive, then confirm SSI is working: + +1. **Check for init containers.** Run this command against one of your application pods. The output should include `datadog-lib--init`: + ```shell + kubectl get pod -n -o jsonpath='{.spec.initContainers[*].name}' + ``` + +2. **Generate traffic.** If your application does not receive traffic automatically, port-forward and send a few requests: + ```shell + kubectl port-forward deployment/ 8080: -n + curl http://localhost:8080/ + ``` + +3. **Find your service.** Go to [**APM** > **Services**][42] and confirm your service appears. + +4. **Confirm traces.** Go to [**APM** > **Traces**][43] and search for traces from your service. + +If traces do not appear after five minutes, see the [SSI troubleshooting guide][35]. + +{{< agent-only >}} +How SSI works (use this to reason about failures): +1. Admission webhook (registered by Cluster Agent) intercepts pod creation +2. Webhook mutates the pod spec — adds a `datadog-lib--init` init container +3. Init container copies the tracer library onto a shared volume +4. `LD_PRELOAD` env var is set pointing to the library .so file +5. Application process loads the library automatically on startup + +What each diagnostic layer can see: +- pup sees what Datadog's backend received. If pup shows no instrumented pods, the problem is in the cluster. +- kubectl sees cluster state. If kubectl shows the init container but pup shows no traces, the problem is post-injection. + +pup CLI setup (if not already available): +- Install: `brew tap datadog-labs/pack && brew install pup` +- Authenticate: `pup auth login` (opens browser for OAuth) +- Verify: `pup auth status` + +After the user restarts their pods, verify with these checks: + +1. Init containers: `kubectl get pod -n -o jsonpath='{.spec.initContainers[*].name}'` + Expected: includes `datadog-lib--init` + +2. If pup is available: + - `pup fleet instrumented-pods list ` + - `pup apm services list --env ` + - `pup traces search --query "service:" --from 1h --limit 5` + - `pup apm service-library-config get --service-name --env ` (validates tracer config) + +3. Runtime version (if compatibility is suspected): + - `kubectl exec -n -- python --version` + - `kubectl exec -n -- node --version` + - `kubectl exec -n -- java -version` + +No init container (injection never happened) — check: +- Namespace targeting: `kubectl get datadogagent datadog -n -o yaml | grep -A 15 instrumentation` +- Pod annotations: `kubectl get pod -n -o yaml | grep -A 5 annotations` +- Pod labels: `kubectl get pod -n --show-labels` +- Admission webhook registered: `kubectl get mutatingwebhookconfigurations | grep datadog` +- Cluster Agent running: `kubectl get pods -n -l app=datadog-cluster-agent` +- Whether the pod is in the Agent namespace (SSI skips its own namespace) + +Init container present but no traces (tracer not reporting) — check: +- Existing instrumentation in application code or dependency manifests +- Agent APM receiver: `kubectl exec -n $(kubectl get pod -n -l app=datadog-agent -o name | head -1) -- agent status | grep -A 5 "APM Agent"` +- Injection errors: `kubectl get pod -n -o jsonpath='{.spec.nodeName}'` then `pup apm troubleshooting list --hostname --timeframe 1h` +- Tracer status: `pup fleet tracers list --filter "service:"` +- Agent connectivity: `pup fleet agents list --filter "hostname:"` +{{< /agent-only >}} + +## Configure Unified Service Tags + +[Unified Service Tags][5] (USTs) apply consistent `service`, `env`, and `version` tags across traces, metrics, and logs. SSI can automatically extract UST values from your existing Kubernetes labels. + +
+Automatic label extraction is not compatible with Remote Configuration. If you use Remote Configuration, configure USTs with ddTraceConfigs instead. +
+ +### Automatic label extraction (recommended) + +Map your Kubernetes labels to Datadog service tags with `kubernetesResourcesLabelsAsTags`. Replace `app.kubernetes.io/name` with whatever label contains your service name: ```yaml datadog: - # Automatically extract service names from Kubernetes labels kubernetesResourcesLabelsAsTags: pods: - app.kubernetes.io/name: service # Modern Kubernetes label + app.kubernetes.io/name: service deployments.apps: app.kubernetes.io/name: service replicasets.apps: app.kubernetes.io/name: service - - # Set environment globally for the entire cluster tags: - "env:production" - apm: instrumentation: enabled: true ``` -With this configuration, Datadog automatically sets the `service` tag using the value of the `app.kubernetes.io/name` label for any instrumented workload that includes this label. +Requires `datadog-agent` 7.69+, `datadog-operator` 1.16.0+, or `datadog-helm-chart` 3.120.0+. -### Configure USTs explicitly with ddTraceConfigs +{{% collapse-content title="Configure USTs with ddTraceConfigs" level="h3" expanded=false %}} -In most cases, automatic configuration is sufficient. However, if you need granular control over settings for specific workloads, use `ddTraceConfigs` to explicitly map labels to service configurations: +For granular control over specific workloads, use `ddTraceConfigs` to map labels to service configurations: ```yaml datadog: @@ -111,11 +278,8 @@ datadog: app.kubernetes.io/name: service deployments.apps: app.kubernetes.io/name: service - - # Set environment globally for the entire cluster tags: - "env:production" - apm: instrumentation: enabled: true @@ -125,137 +289,57 @@ datadog: matchLabels: tier: frontend ddTraceConfigs: - - name: DD_SERVICE # Explicitly override service name + - name: DD_SERVICE valueFrom: fieldRef: fieldPath: metadata.labels['app.kubernetes.io/name'] - # DD_ENV inherited from cluster-level tags above - # DD_VERSION automatically extracted from image tags ``` +{{% /collapse-content %}} -### Configure USTs in deployment manifests +{{% collapse-content title="Configure USTs in deployment manifests" level="h3" expanded=false %}} -If your setup doesn't use labels suitable for UST extraction, you can set USTs directly in your deployment manifests using environment variables. This approach requires modifying each deployment individually, but offers precise control. +If your labels are not suitable for automatic extraction, set USTs directly in your deployment manifests with environment variables. This requires modifying each deployment individually. For complete instructions, see [setting USTs for Kubernetes services][5]. -## Enable SDK-dependent products and features - -After SSI loads the Datadog SDK into your applications and enables distributed tracing, you can configure additional products that rely on the SDK: - -{{< ssi-products >}} - -Use one of the following setup methods: - -- **[Configure with workload targeting (recommended)](#target-specific-workloads)**: - - By default, Single Step Instrumentation instruments all services in all namespaces. Use workload targeting to limit instrumentation to specific namespaces, pods, or workloads, and apply custom configurations. - -- **[Set environment variables][7]**: - - Enable products by setting environment variables directly in your application configuration. - -## Advanced options - -Use the following advanced options to customize how Single Step Instrumentation behaves in your environment. These settings are optional and typically only needed in specialized setups. - -### Configure injection modes - -SSI supports multiple injection modes, which control how the injector and APM library files are delivered to your application containers. You typically do not need to configure this setting manually. Consider adjusting it if you notice significant pod startup delays or higher-than-expected resource usage (CPU, memory) during pod initialization. For more on how the injector works, see [Injector Behavior with Single Step Instrumentation][41]. - - -| Mode | Description | Requirements | -|------|-------------|--------------| -| `init_container` | Uses init containers to copy injector and APM library files into application containers. | Agent deployed with Helm Chart or Datadog Operator | -| `csi` | **In Preview.** Mounts injector and APM library files using the [Datadog CSI driver][37]. Reduces pod startup time compared to init container mode. | Agent 7.76.0+, CSI driver 1.2.0+, Helm Chart 3.178.1+ or Datadog Operator 1.25.0+ | - -Before using `csi` mode, install and activate the Datadog CSI driver. If you are deploying with Helm, also set `datadog.csi.enabled: true` in your `datadog-values.yaml`. See the [CSI driver documentation][37] for installation steps and environment-specific requirements such as GKE Autopilot. - -#### Configure injection mode globally - -{{< tabs >}} -{{% tab "Helm" %}} - -To set the injection mode cluster-wide, add `injectionMode` to your `datadog-values.yaml`: - -```yaml -datadog: - apm: - instrumentation: - injectionMode: -``` - -Supported values: `init_container`, `csi`. - -{{% /tab %}} -{{% tab "Datadog Operator" %}} - -To set the injection mode cluster-wide, add `injectionMode` to your `datadog-agent.yaml`: - -```yaml -features: - apm: - instrumentation: - injectionMode: -``` - -Supported values: `init_container`, `csi`. +{{% /collapse-content %}} -If you are using Datadog Operator earlier than 1.25.0, use the [pod annotation](#configure-injection-mode-per-pod) to override the injection mode for specific pods. +## Next steps -{{% /tab %}} -{{< /tabs >}} +After traces are flowing: -#### Configure injection mode per pod +{{< whatsnext desc=" " >}} + {{< nextlink href="/tracing/trace_explorer/" >}}Trace Explorer: search and analyze your traces{{< /nextlink >}} + {{< nextlink href="/tracing/services/service_page/" >}}Service Page: monitor service health and performance{{< /nextlink >}} + {{< nextlink href="/tracing/trace_collection/custom_instrumentation/" >}}Custom instrumentation: add application-specific spans{{< /nextlink >}} + {{< nextlink href="/dynamic_instrumentation/" >}}Dynamic Instrumentation: add custom spans without redeploying{{< /nextlink >}} +{{< /whatsnext >}} -To override the injection mode for a specific pod, add the following annotation to the pod spec: - -```yaml -metadata: - annotations: - admission.datadoghq.com/apm-inject.injection-mode: "" -``` - -Supported values: `init_container`, `csi`. +## Advanced SSI configuration ### Target specific workloads -By default, SSI instruments all services in all namespaces in your cluster. Depending on your Agent version, use one of the following configuration methods to refine which services are instrumented and how. +By default, SSI instruments all services in all namespaces. Use one of the following methods to limit instrumentation scope. {{< tabs >}} {{% tab "Agent v7.64+ (Recommended)" %}} -Create targeting blocks with the `targets` label to specify which workloads to instrument and what configurations to apply. - -Each target block has the following keys: +Define targeting blocks with `targets` to control which workloads are instrumented and what configuration they receive. | Key | Description | |------------------|-------------| -| `name` | The name of the target block. This has no effect on monitoring state and is used only as metadata. | -| `namespaceSelector` | The namespace(s) to instrument. Specify using one or more of:
- `matchNames`: A list of one or more namespace name(s).
- `matchLabels`: A list of one or more label(s) defined in `{key,value}` pairs.
- `matchExpressions`: A list of namespace selector requirements.

Namespaces must meet all criteria to match. For more details, see the [Kubernetes selector documentation][10].| -| `podSelector` | The pod(s) to instrument. Specify using one or more of:
- `matchLabels`: A list of one or more label(s) defined in `{key,value}` pairs.
- `matchExpressions`: A list of pod selector requirements.

Pods must meet all criteria to match. For more details, see the [Kubernetes selector documentation][10]. | -| `ddTraceVersions` | The [Datadog APM SDK][9] version to use for each language. | -| `ddTraceConfigs` | APM SDK configs that allow setting [Unified Service Tags][8], enabling [SDK-dependent products](#enable-sdk-dependent-products-and-features) beyond tracing, and customizing other [APM settings][14]. | - -The file you need to configure depends on how you enabled Single Step Instrumentation: -- If you enabled SSI with Datadog Operator, edit `datadog-agent.yaml`. -- If you enabled SSI with Helm, edit `datadog-values.yaml`. - -**Note**: Targets are evaluated in order; the first match takes precedence. - -#### Example configurations +| `name` | Target block name (metadata only). | +| `namespaceSelector` | Namespace(s) to instrument. Use `matchNames`, `matchLabels`, or `matchExpressions`. See the [Kubernetes selector documentation][10].| +| `podSelector` | Pod(s) to instrument. Use `matchLabels` or `matchExpressions`. See the [Kubernetes selector documentation][10]. | +| `ddTraceVersions` | [Datadog APM SDK][9] version per language. | +| `ddTraceConfigs` | SDK configuration: [Unified Service Tags][8], [additional products](#enable-additional-products), and [other APM settings][14]. | -Review the following examples demonstrating how to select specific services: +Edit `datadog-values.yaml` (Helm) or `datadog-agent.yaml` (Operator). Targets are evaluated in order; the first match wins. {{< collapse-content title="Example 1: Enable all namespaces except one" level="h4" >}} -This configuration: -- enables APM for all namespaces except the `jenkins` namespace. - - **Note**: use `enabledNamespaces` to disable for all namespaces except those listed. -- instructs Datadog to instrument the Java applications with the default Java SDK and Python applications with `v.3.1.0` of the Python SDK. - {{< highlight yaml "hl_lines=4-10" >}} apm: instrumentation: @@ -271,17 +355,7 @@ This configuration: {{< /collapse-content >}} -{{< collapse-content title="Example 2: Instrument a subset of namespaces, matching on names and labels" level="h4" >}} - -This configuration creates two targets blocks: - -- The first block (named `login-service_namespace`): - - enables APM for services in the namespace `login-service`. - - instructs Datadog to instrument services in this namespace with the default version of the Java SDK. - - sets environment variable `DD_PROFILING_ENABLED` for this target group -- The second block (named `billing-service_apps`) - - enables APM for services in the namespace(s) with label `app:billing-service`. - - instructs Datadog to instrument this set of services with `v3.1.0` of the Python SDK. +{{< collapse-content title="Example 2: Instrument specific namespaces by name and label" level="h4" >}} {{< highlight yaml "hl_lines=4-28" >}} apm: @@ -295,7 +369,7 @@ This configuration creates two targets blocks: ddTraceVersions: java: "default" ddTraceConfigs: - - name: "DD_PROFILING_ENABLED" ## profiling is enabled for all services in this namespace + - name: "DD_PROFILING_ENABLED" value: "auto" - name: "billing-service_apps" namespaceSelector: @@ -307,14 +381,7 @@ This configuration creates two targets blocks: {{< /collapse-content >}} -{{< collapse-content title="Example 3: Instrument different workloads with different tracers" level="h4" >}} - -This configuration does the following: -- enables APM for pods with the following labels: - - `app:db-user`, which marks pods running the `db-user` application. - - `webserver:routing`, which marks pods running the `request-router` application. -- instructs Datadog to use the default versions of the Datadog Tracer SDKs. -- sets Datadog environment variables to apply to each target group and configure the SDKs. +{{< collapse-content title="Example 3: Different SDKs for different workloads" level="h4" >}} {{< highlight yaml "hl_lines=4-28" >}} apm: @@ -327,7 +394,7 @@ This configuration does the following: app: "db-user" ddTraceVersions: java: "default" - ddTraceConfigs: ## trace configs set for services in matching pods + ddTraceConfigs: - name: "DD_DATA_STREAMS_ENABLED" value: "true" - name: "user-request-router" @@ -340,12 +407,7 @@ This configuration does the following: {{< /collapse-content >}} -{{< collapse-content title="Example 4: Instrument a pod within a namespace" level="h4" >}} - -This configuration: -- enables APM for pods labeled `app:password-resolver` inside the `login-service` namespace. -- instructs Datadog to use the default version of the Datadog Java Tracer SDK. -- sets Datadog environment variables to apply to this target. +{{< collapse-content title="Example 4: Target a pod within a namespace" level="h4" >}} {{< highlight yaml "hl_lines=4-28" >}} apm: @@ -368,9 +430,7 @@ This configuration: {{< /collapse-content >}} -{{< collapse-content title="Example 5: Instrument a subset of pods using matchExpressions" level="h4" >}} - -This configuration enables APM for all pods except those that have either of the labels `app=app1` or `app=app2`. +{{< collapse-content title="Example 5: Exclude specific pods with matchExpressions" level="h4" >}} {{< highlight yaml "hl_lines=4-28" >}} apm: @@ -389,9 +449,9 @@ This configuration enables APM for all pods except those that have either of the {{< /collapse-content >}} -{{< collapse-content title="Example 6: Enable additional products with ddTraceConfigs" level="h4" >}} +{{< collapse-content title="Example 6: Enable additional products" level="h4" >}} -This configuration enables [App and API Protection (AAP)][12] and [Continuous Profiler][11] for services in the `web-apps` namespace, using `ddTraceConfigs` to set the required environment variables: +Enable [App and API Protection][12] and [Continuous Profiler][11] for a namespace: {{< highlight yaml "hl_lines=4-20" >}} apm: @@ -412,8 +472,6 @@ This configuration enables [App and API Protection (AAP)][12] and [Continuous Pr value: "auto" {{< /highlight >}} -For a full list of products you can enable through SSI, see [Enable SDK-dependent products and features](#enable-sdk-dependent-products-and-features). - {{< /collapse-content >}} [8]: /getting_started/tagging/unified_service_tagging/?tab=kubernetes @@ -427,34 +485,18 @@ For a full list of products you can enable through SSI, see [Enable SDK-dependen {{% tab "Agent <=v7.63 (Legacy)" %}} -#### Enable or disable instrumentation for namespaces - -You can choose to enable or disable instrumentation for applications in specific namespaces. You can only set enabledNamespaces or disabledNamespaces, not both. +#### Namespace filtering -The file you need to configure depends on if you enabled Single Step Instrumentation with Datadog Operator or Helm: +Enable or disable instrumentation for specific namespaces. You can set `enabledNamespaces` or `disabledNamespaces`, but not both. {{< collapse-content title="Datadog Operator" level="h5" >}} -To enable instrumentation for specific namespaces, add `enabledNamespaces` configuration to `datadog-agent.yaml`: - -{{< highlight yaml "hl_lines=5-7" >}} - features: - apm: - instrumentation: - enabled: true - enabledNamespaces: # Add namespaces to instrument - - default - - applications -{{< /highlight >}} - -To disable instrumentation for specific namespaces, add `disabledNamespaces` configuration to `datadog-agent.yaml`: - {{< highlight yaml "hl_lines=5-7" >}} features: apm: instrumentation: enabled: true - disabledNamespaces: # Add namespaces to not instrument + enabledNamespaces: - default - applications {{< /highlight >}} @@ -463,101 +505,51 @@ To disable instrumentation for specific namespaces, add `disabledNamespaces` con {{< collapse-content title="Helm" level="h5" >}} -To enable instrumentation for specific namespaces, add `enabledNamespaces` configuration to `datadog-values.yaml`: - {{< highlight yaml "hl_lines=5-7" >}} datadog: apm: instrumentation: enabled: true - enabledNamespaces: # Add namespaces to instrument + enabledNamespaces: - namespace_1 - namespace_2 {{< /highlight >}} -To disable instrumentation for specific namespaces, add `disabledNamespaces` configuration to `datadog-values.yaml`: - -{{< highlight yaml "hl_lines=5-7" >}} - datadog: - apm: - instrumentation: - enabled: true - disabledNamespaces: # Add namespaces to not instrument - - namespace_1 - - namespace_2 -{{< /highlight >}} - {{< /collapse-content >}} -#### Specify SDK versions - -
Starting with Datadog Cluster Agent v7.52.0+, you can automatically instrument a subset of your applications, based on the SDKs you specify.
- -Specify Datadog SDKs and their versions to automatically instrument applications written in those languages. You can configure this in two ways, which are applied in the following order of precedence: +#### SDK versions -1. [Specify at the service level](#specify-at-the-service-level), or -2. [Specify at the cluster level](#specify-at-the-cluster-level). +Specify SDK versions to control which languages are instrumented and which library versions are used. You can set versions at the [service level](#specify-at-the-service-level) (pod annotations) or at the [cluster level](#specify-at-the-cluster-level) (Agent config). Service-level settings take precedence. -**Default**: If you don't specify any library versions, applications written in supported languages are automatically instrumented using the latest SDK versions. +If you don't specify versions, all supported languages are instrumented with the latest SDK. ##### Specify at the service level -To automatically instrument applications in specific pods, add the appropriate language annotation and library version for your application in your pod spec: +Add a language annotation to your pod spec: | Language | Pod annotation | |------------|-----------------------------------------------------------------------| -| Java | `admission.datadoghq.com/java-lib.version: ""` | -| Node.js | `admission.datadoghq.com/js-lib.version: ""` | -| Python | `admission.datadoghq.com/python-lib.version: ""` | -| .NET | `admission.datadoghq.com/dotnet-lib.version: ""` | -| Ruby | `admission.datadoghq.com/ruby-lib.version: ""` | -| PHP | `admission.datadoghq.com/php-lib.version: ""` | +| Java | `admission.datadoghq.com/java-lib.version: ""` | +| Node.js | `admission.datadoghq.com/js-lib.version: ""` | +| Python | `admission.datadoghq.com/python-lib.version: ""` | +| .NET | `admission.datadoghq.com/dotnet-lib.version: ""` | +| Ruby | `admission.datadoghq.com/ruby-lib.version: ""` | +| PHP | `admission.datadoghq.com/php-lib.version: ""` | -Replace `` with the desired library version. Available versions are listed in the [Datadog container registries](#change-the-default-image-registry) and tracer source repositories for each language: +Available versions: [Java][34], [Node.js][35], [Python][36], [.NET][37], [Ruby][38], [PHP][39]. -- [Java][34] -- [Node.js][35] -- [Python][36] -- [.NET][37] -- [Ruby][38] -- [PHP][39] - -
Exercise caution when using the latest tag, as major library releases may introduce breaking changes.
- -For example, to automatically instrument Java applications: - -{{< highlight yaml "hl_lines=10" >}} -apiVersion: apps/v1 -kind: Deployment -metadata: - labels: - # ... -spec: - template: - metadata: - annotations: - admission.datadoghq.com/java-lib.version: "" - spec: - containers: - - # ... -{{< /highlight >}} +
Exercise caution with the latest tag. Major releases may introduce breaking changes.
##### Specify at the cluster level -If you don't enable automatic instrumentation for specific pods using annotations, you can specify which languages to instrument across the entire cluster using the SSI configuration. When `apm.instrumentation.libVersions` is set, only applications written in the specified languages are instrumented, using the specified library versions. - -The file you need to configure depends on if you enabled Single Step Instrumentation with Datadog Operator or Helm: - {{< collapse-content title="Datadog Operator" level="h5" >}} -For example, to instrument .NET, Python, and Node.js applications, add the following configuration to your `datadog-agent.yaml` file: - {{< highlight yaml "hl_lines=5-8" >}} features: apm: instrumentation: enabled: true - libVersions: # Add any libraries and versions you want to set + libVersions: dotnet: "x.x.x" python: "x.x.x" js: "x.x.x" @@ -567,14 +559,12 @@ For example, to instrument .NET, Python, and Node.js applications, add the follo {{< collapse-content title="Helm" level="h5" >}} -For example, to instrument .NET, Python, and Node.js applications, add the following configuration to your `datadog-values.yaml` file: - {{< highlight yaml "hl_lines=5-8" >}} datadog: apm: instrumentation: enabled: true - libVersions: # Add any libraries and versions you want to set + libVersions: dotnet: "x.x.x" python: "x.x.x" js: "x.x.x" @@ -582,7 +572,6 @@ For example, to instrument .NET, Python, and Node.js applications, add the follo {{< /collapse-content >}} - [34]: https://github.com/DataDog/dd-trace-java/releases [35]: https://github.com/DataDog/dd-trace-js/releases [36]: https://github.com/DataDog/dd-trace-py/releases @@ -590,264 +579,189 @@ For example, to instrument .NET, Python, and Node.js applications, add the follo [38]: https://github.com/DataDog/dd-trace-rb/releases [39]: https://github.com/DataDog/dd-trace-php/releases - {{% /tab %}} {{< /tabs >}} -### Change the default image registry +### Enable additional products -Datadog publishes instrumentation libraries images on gcr.io, Docker Hub, and Amazon ECR: +After SSI enables distributed tracing, you can activate additional SDK-dependent products: -| Language | gcr.io | hub.docker.com | gallery.ecr.aws | -|------------|-------------------------------------|---------------------------------------------|-------------------------------------------| -| Java | [gcr.io/datadoghq/dd-lib-java-init][15] | [hub.docker.com/r/datadog/dd-lib-java-init][16] | [gallery.ecr.aws/datadog/dd-lib-java-init][17] | -| Node.js | [gcr.io/datadoghq/dd-lib-js-init][18] | [hub.docker.com/r/datadog/dd-lib-js-init][19] | [gallery.ecr.aws/datadog/dd-lib-js-init][20] | -| Python | [gcr.io/datadoghq/dd-lib-python-init][21] | [hub.docker.com/r/datadog/dd-lib-python-init][22] | [gallery.ecr.aws/datadog/dd-lib-python-init][23] | -| .NET | [gcr.io/datadoghq/dd-lib-dotnet-init][24] | [hub.docker.com/r/datadog/dd-lib-dotnet-init][25] | [gallery.ecr.aws/datadog/dd-lib-dotnet-init][26] | -| Ruby | [gcr.io/datadoghq/dd-lib-ruby-init][27] | [hub.docker.com/r/datadog/dd-lib-ruby-init][28] | [gallery.ecr.aws/datadog/dd-lib-ruby-init][29] | -| PHP | [gcr.io/datadoghq/dd-lib-php-init][30] | [hub.docker.com/r/datadog/dd-lib-php-init][31] | [gallery.ecr.aws/datadog/dd-lib-php-init][32] | +{{< ssi-products >}} -The `DD_ADMISSION_CONTROLLER_AUTO_INSTRUMENTATION_CONTAINER_REGISTRY` environment variable in the Datadog Cluster Agent configuration specifies the registry used by the Admission Controller. The default value is `gcr.io/datadoghq`. +To enable products: -You can pull the SDK from a different registry by changing it to `docker.io/datadog`, `public.ecr.aws/datadog`, or another URL if you are hosting the images in a local container registry. +- **With workload targeting (recommended):** Add `ddTraceConfigs` entries to your [target blocks](#target-specific-workloads). See [Example 6: Enable additional products](#target-specific-workloads). +- **With environment variables:** Set variables directly in your application configuration. See [Library Configuration][7]. -For instructions on changing your container registry, see [Changing Your Container Registry][33]. +### Configure injection modes -### Use a private container registry +SSI supports multiple injection modes that control how library files are delivered to application containers. Adjust this setting if you notice pod startup delays or high resource usage during initialization. -If your organization does not allow direct pulls from public registries (such as `gcr.io`, `docker.io`, or `public.ecr.aws`), you can host the required Datadog images internally and configure the Admission Controller to use them. +| Mode | Description | Requirements | +|------|-------------|--------------| +| `init_container` | Copies files with an init container. | Agent deployed with Helm or Operator | +| `csi` | **In Preview.** Mounts files with the [Datadog CSI driver][37]. Faster pod startup. | Agent 7.76.0+, CSI driver 1.2.0+, Helm 3.178.1+ or Operator 1.25.0+ | -To use SSI with a private container registry: +For `csi` mode, install and activate the CSI driver first. With Helm, set `datadog.csi.enabled: true`. See the [CSI driver documentation][37]. -1. Follow [these instructions][34] to mirror Datadog's container images to your private registry. +{{< collapse-content title="Set injection mode globally" level="h4" >}} - You only need the images for the languages you are instrumenting. If you're not sure which ones you need, here's a baseline that covers most use cases: +**Helm** — add to `datadog-values.yaml`: +```yaml +datadog: + apm: + instrumentation: + injectionMode: +``` - - `apm-inject` - - `dd-lib-java-init` - - `dd-lib-python-init` - - `dd-lib-dotnet-init` - - `dd-lib-php-init` - - `dd-lib-ruby-init` - - `dd-lib-js-init` +**Datadog Operator** — add to `datadog-agent.yaml`: +```yaml +features: + apm: + instrumentation: + injectionMode: +``` - You can find these images on [gcr.io][12], [Docker Hub][13], or [Amazon ECR Public Gallery][14]. +Supported values: `init_container`, `csi`. -2. Tag the images according to your configuration. +{{< /collapse-content >}} - The versions you mirror must match the versions configured in your workloads, which might be set in one of the following ways: - - globally in the Agent config using `ddTraceVersions`, or - - per-pod using annotations like `admission.datadoghq.com/java-lib.version`. +{{< collapse-content title="Set injection mode per pod" level="h4" >}} - If no version is explicitly configured, the default version (`0`) is used. +Add this annotation to the pod spec: - For example: +```yaml +metadata: + annotations: + admission.datadoghq.com/apm-inject.injection-mode: "" +``` - ``` - apm: - instrumentation: - enabled: true - targets: - - name: "default-target" - ddTraceVersions: - java: "1" - python: "3" - ``` +Supported values: `init_container`, `csi`. - This configuration requires the following image tags: - - `apm-inject:0` - - `dd-lib-java-init:1` - - `dd-lib-python-init:3` +{{< /collapse-content >}} -3. Update the Cluster Agent configuration to use your private registry. +### Change the image registry - Set the `DD_ADMISSION_CONTROLLER_AUTO_INSTRUMENTATION_CONTAINER_REGISTRY` environment variable in your Cluster Agent config to use your private registry. +Datadog publishes SDK images on gcr.io (default), Docker Hub, and Amazon ECR. To use a different registry, set `DD_ADMISSION_CONTROLLER_AUTO_INSTRUMENTATION_CONTAINER_REGISTRY` in the Cluster Agent config: -For more details on changing your container registry, see [Changing Your Container Registry][33]. +| Registry | Value | +|----------|-------| +| gcr.io (default) | `gcr.io/datadoghq` | +| Docker Hub | `docker.io/datadog` | +| Amazon ECR | `public.ecr.aws/datadog` | -### Using a Container Network Interface on EKS +For detailed instructions, see [Changing Your Container Registry][33]. -When using a CNI like Calico, the control plane nodes are not able to initiate network connections to Datadog's Admission Controller and report an "Address is not allowed" error. -To use Single Step instrumentation, modify Datadog's Cluster Agent with the `useHostNetwork: true` parameter. +{{% collapse-content title="Use a private container registry" level="h4" expanded=false %}} -``` -datadog: - ... +If your organization cannot pull from public registries, mirror the Datadog images to your private registry: -clusterAgent: - useHostNetwork: true +1. [Mirror the images][34] for the languages you are instrumenting. At minimum, mirror `apm-inject` and the `dd-lib--init` images you need. - admissionController: - ... -``` +2. Tag the images to match your configuration. If you set `ddTraceVersions` to `java: "1"` and `python: "3"`, mirror: + - `apm-inject:0` + - `dd-lib-java-init:1` + - `dd-lib-python-init:3` -## Remove Single Step APM instrumentation from your Agent +3. Set `DD_ADMISSION_CONTROLLER_AUTO_INSTRUMENTATION_CONTAINER_REGISTRY` to your private registry URL. -If you don't want to collect trace data for a particular service, host, VM, or container, complete the following steps: +{{% /collapse-content %}} -### Remove instrumentation for specific services +### Container Network Interface on EKS -To remove APM instrumentation and stop sending traces from a specific service, you can do one of the following: +When using a CNI like Calico on EKS, control plane nodes cannot connect to the Admission Controller. Set `useHostNetwork: true` on the Cluster Agent: -#### Use instrumentation rules to target specific workloads (recommended) +```yaml +clusterAgent: + useHostNetwork: true +``` -With instrumentation rules (available for Agent v7.64+), you can enable and disable tracing for specific applications. [See configuration details here](#advanced-options). +## Remove SSI -#### Use the Datadog Admission Controller +### Remove instrumentation for specific services -As an alternative, or for a version of the agent that does not support instrumentation rules, you can also disable pod mutation by adding a label to your pod. +Use [workload targeting](#target-specific-workloads) (Agent v7.64+) to exclude specific services. -
In addition to disabling SSI, the following steps disable other mutating webhooks. Use with caution.
+Alternatively, add this label to the pod spec to skip Admission Controller mutation: -1. Set the `admission.datadoghq.com/enabled:` label to `"false"` for the pod spec: - ```yaml - spec: - template: - metadata: - labels: - admission.datadoghq.com/enabled: "false" - ``` -2. Apply the configuration: - ```shell - kubectl apply -f /path/to/your/deployment.yaml - ``` -3. Restart the services you want to remove instrumentation for. +
This label disables all mutating webhooks for the pod, not only SSI.
-### Remove APM for all services on the infrastructure +```yaml +spec: + template: + metadata: + labels: + admission.datadoghq.com/enabled: "false" +``` -To stop producing traces, uninstall APM and restart the infrastructure: +Apply the change and restart the affected pods. -The file you need to configure depends on if you enabled Single Step Instrumentation with Datadog Operator or Helm: +### Remove instrumentation for all services {{< tabs >}} -{{% tab "Datadog Operator" %}} +{{% tab "Helm" %}} -1. Set `instrumentation.enabled=false` in `datadog-agent.yaml`: - ```yaml - features: - apm: - instrumentation: - enabled: false - ``` +Set `instrumentation.enabled: false` in `datadog-values.yaml` and run: + +```shell +helm upgrade datadog-agent -f datadog-values.yaml datadog/datadog +``` -2. Deploy the Datadog Agent with the updated configuration file: - ```shell - kubectl apply -f /path/to/your/datadog-agent.yaml - ``` {{% /tab %}} +{{% tab "Datadog Operator" %}} -{{% tab "Helm" %}} +Set `instrumentation.enabled: false` in `datadog-agent.yaml` and apply: -1. Set `instrumentation.enabled=false` in `datadog-values.yaml`: - ```yaml - datadog: - apm: - instrumentation: - enabled: false - ``` +```shell +kubectl apply -f datadog-agent.yaml +``` -2. Run the following command: - ```shell - helm upgrade datadog-agent -f datadog-values.yaml datadog/datadog - ``` {{% /tab %}} {{< /tabs >}} -## Best practices - -After you enable SSI, all supported processes in the cluster are automatically instrumented and begin producing traces within minutes. +## SSI best practices -To control where APM is activated and reduce overhead, consider the following best practices. +{{% collapse-content title="Use opt-in labels for controlled rollout" level="h3" expanded=false %}} -{{% collapse-content title="Use opt-in labels for controlled APM rollout" level="h3" expanded=false id="id-for-anchoring" %}} - -#### Default vs. opt-in instrumentation | Mode | Behavior | When to use | | --- | ----------- | ----------- | -| Default | All supported processes in the cluster are instrumented. | Small clusters or prototypes. | -| Opt-in | Use [instrumentation rules][4] to restrict instrumentation to specific namespaces or pods. | Production clusters, staged rollouts, or cost‑sensitive use cases. | - -#### Example: Enable instrumentation for specific pods +| Default | All supported processes are instrumented. | Small clusters or prototypes. | +| Opt-in | [Workload targeting](#target-specific-workloads) restricts instrumentation to labeled pods. | Production clusters, staged rollouts, cost-sensitive environments. | -1. Add a meaningful label (for example, `datadoghq.com/apm-instrumentation: "enabled"`) to both the deployment metadata and the pod template. +Add an opt-in label to your deployment and pod template: - ``` - apiVersion: apps/v1 - kind: Deployment - metadata: - name: checkout-api - labels: - app: checkout-api - datadoghq.com/apm-instrumentation: "enabled" # opt-in label (cluster-wide) - spec: - replicas: 3 - selector: - matchLabels: - app: checkout-api - template: - metadata: - labels: - app: checkout-api - datadoghq.com/apm-instrumentation: "enabled" # opt-in label must be on *template*, too - # Unified Service Tags (recommended) - tags.datadoghq.com/service: "checkout-api" - tags.datadoghq.com/env: "prod" - tags.datadoghq.com/version: "2025-06-10" - spec: - containers: - - name: api - image: my-registry/checkout:latest - ports: - - containerPort: 8080 - ``` - -2. In your Datadog Agent Helm config, enable SSI and use `podSelector` to inject only into pods with the matching opt-in label. +```yaml +metadata: + labels: + datadoghq.com/apm-instrumentation: "enabled" +``` - ``` - apm: - instrumentation: - enabled: true - targets: - - name: apm-instrumented - podSelector: - matchLabels: - datadoghq.com/apm-instrumentation: "enabled" - ``` +Then configure SSI to match: -See [instrumentation rules][4] for additional examples. +```yaml +apm: + instrumentation: + enabled: true + targets: + - name: apm-instrumented + podSelector: + matchLabels: + datadoghq.com/apm-instrumentation: "enabled" +``` {{% /collapse-content %}} +{{% collapse-content title="Pin SDK versions" level="h3" expanded=false %}} -{{% collapse-content title="Control which Datadog SDKs are loaded" level="h3" expanded=false id="id-for-anchoring" %}} - -Use `ddTraceVersions` in your Agent Helm config to control both the language and the version of the Datadog SDK. This prevents unnecessary SDKs from being downloaded, which minimizes init-container footprint, reduces image size, and allows for more deliberate tracer upgrades (for example, to meet compliance requirements or simplify debugging). +Use `ddTraceVersions` to control which SDK versions are injected. This reduces init container size, avoids downloading unnecessary SDKs, and makes tracer upgrades deliberate. -#### Example: Specify a Java SDK for a namespace - -Only Java applications run in the `login-service` namespace. To avoid downloading other SDKs, configure the Agent to target that namespace and inject only the Java SDK version 1.48.2. - - -``` -targets: - - name: login-service - namespaceSelector: - matchNames: ["login-service"] - ddTraceVersions: - java: "1.48.2" # pin version -``` - -#### Default configuration - -If a pod doesn't match any `ddTraceVersions` rule, the default target applies. - -``` +```yaml targets: - - name: default-target # tag any pod *without* an override + - name: default-target ddTraceVersions: - java: "1" # stay on latest v1.x - python: "3" # stay on latest v3.x - js: "5" # NodeJS + java: "1" + python: "3" + js: "5" php: "1" dotnet: "3" ``` @@ -856,7 +770,7 @@ targets: ## Troubleshooting -If you encounter problems enabling APM with SSI, see the [SSI troubleshooting guide][35]. +If you encounter problems with SSI, see the [SSI troubleshooting guide][35]. ## Further reading @@ -865,34 +779,14 @@ If you encounter problems enabling APM with SSI, see the [SSI troubleshooting gu [1]: https://v3.helm.sh/docs/intro/install/ [2]: https://kubernetes.io/docs/tasks/tools/install-kubectl/ [3]: /tracing/glossary/#instrumentation -[4]: /tracing/trace_collection/automatic_instrumentation/single-step-apm/kubernetes/?tab=agentv764recommended#configure-instrumentation-for-namespaces-and-pods [5]: /getting_started/tagging/unified_service_tagging/?tab=kubernetes#containerized-environment [7]: /tracing/trace_collection/library_config/ [11]: https://app.datadoghq.com/fleet/install-agent/latest?platform=kubernetes -[12]: https://gcr.io/datadoghq -[13]: https://hub.docker.com/u/datadog -[14]: https://gallery.ecr.aws/datadog -[15]: http://gcr.io/datadoghq/dd-lib-java-init -[16]: http://hub.docker.com/r/datadog/dd-lib-java-init -[17]: http://gallery.ecr.aws/datadog/dd-lib-java-init -[18]: http://gcr.io/datadoghq/dd-lib-js-init -[19]: http://hub.docker.com/r/datadog/dd-lib-js-init -[20]: http://gallery.ecr.aws/datadog/dd-lib-js-init -[21]: http://gcr.io/datadoghq/dd-lib-python-init -[22]: http://hub.docker.com/r/datadog/dd-lib-python-init -[23]: http://gallery.ecr.aws/datadog/dd-lib-python-init -[24]: http://gcr.io/datadoghq/dd-lib-dotnet-init -[25]: http://hub.docker.com/r/datadog/dd-lib-dotnet-init -[26]: http://gallery.ecr.aws/datadog/dd-lib-dotnet-init -[27]: http://gcr.io/datadoghq/dd-lib-ruby-init -[28]: http://hub.docker.com/r/datadog/dd-lib-ruby-init -[29]: http://gallery.ecr.aws/datadog/dd-lib-ruby-init -[30]: http://gcr.io/datadoghq/dd-lib-php-init -[31]: http://hub.docker.com/r/datadog/dd-lib-php-init -[32]: http://gallery.ecr.aws/datadog/dd-lib-php-init [33]: /containers/guide/changing_container_registry/ [34]: /containers/guide/sync_container_images/#copy-an-image-to-another-registry-using-crane -[35]: /tracing/trace_collection/automatic_instrumentation/single-step-apm/troubleshooting -[36]: /tracing/trace_collection/automatic_instrumentation/single-step-apm/compatibility/ +[35]: /tracing/trace_collection/single-step-apm/troubleshooting +[36]: /tracing/trace_collection/single-step-apm/compatibility/ [37]: /containers/kubernetes/csi_driver/ -[41]: /tracing/guide/injectors/ +[42]: https://app.datadoghq.com/apm/services +[43]: https://app.datadoghq.com/apm/traces +[44]: /tracing/trace_collection/dd_libraries/nodejs/ diff --git a/content/en/tracing/trace_collection/single-step-apm/troubleshooting.md b/content/en/tracing/trace_collection/single-step-apm/troubleshooting.md index 4b715a49e76..d5182399144 100644 --- a/content/en/tracing/trace_collection/single-step-apm/troubleshooting.md +++ b/content/en/tracing/trace_collection/single-step-apm/troubleshooting.md @@ -15,6 +15,56 @@ further_reading: Single Step Instrumentation (SSI) helps instrument applications by automatically loading application processes with the Datadog SDKs. SSI works for applications running on Linux hosts, in container environments such as Kubernetes and Docker, and for .NET applications served by Windows IIS—without requiring changes to application dependencies or images. If you encounter issues enabling APM with SSI, use this guide to troubleshoot and resolve common problems. For further assistance, contact [Datadog Support][1]. +## Triage: no traces after enabling SSI? + +If you enabled SSI but don't see traces, work through these checks in order: + +1. **Did you restart your pods or processes after enabling SSI?** SSI injects at startup. Existing pods and processes are not instrumented until restarted. + +2. **Does your application have existing tracer dependencies?** SSI silently disables itself if it detects `ddtrace`, `dd-trace`, OpenTelemetry SDK, or `-javaagent` in your application. Check your dependency manifests: + ``` + grep -rn "ddtrace\|dd-trace\|opentelemetry" requirements.txt package.json Gemfile go.mod pom.xml 2>/dev/null + ``` + Remove these dependencies and rebuild your application if found. + +3. **Is your application in the same namespace as the Datadog Agent?** SSI does not instrument pods in the Agent namespace. + +4. **Is a namespace or pod selector filtering your application out?** Check your SSI configuration for `enabledNamespaces`, `disabledNamespaces`, or `podSelector` targets that may not match your application's namespace or labels. Also check for the `admission.datadoghq.com/enabled: "false"` annotation on the pod, which tells the Admission Controller to skip it. + +5. **Is the runtime version supported?** Check the [SSI compatibility guide][13]. + +6. **For Node.js: is your application using ECMAScript Modules (ESM)?** SSI does not support ESM. If your application uses `import` syntax or sets `"type": "module"` in `package.json`, use [manually managed SDKs][14] instead. + +If none of these apply, continue with the detailed troubleshooting methods below. + +{{< agent-only >}} +When diagnosing missing traces after SSI enablement, run these diagnostic commands: + +Triage (run all simultaneously): +- `kubectl get pod -n -o jsonpath='{.spec.initContainers[*].name}'` +- `kubectl describe pod -n | grep -A 10 "Events:"` +- `kubectl get datadogagent datadog -n -o yaml | grep -A 15 instrumentation` +- If pup CLI is available: + - `pup traces search --query "service:" --from 1h --limit 5` + - `pup fleet instrumented-pods list ` + - `pup apm troubleshooting list --hostname --timeframe 1h` + +Diagnosis by signal: +| Signal | Likely cause | Next step | +|--------|-------------|-----------| +| No init container on pod | Injection never happened | Check namespace targeting, pod-selector, opt-out annotation, webhook registration, pod not restarted | +| Init container present, no traces | Injection attempted but tracer not reporting | Check for existing ddtrace/OTel dependencies, runtime version, Agent connectivity | +| Pod events show CrashLoopBackOff | Init container failure | Check existing instrumentation conflicts, runtime version | +| Traces arriving but wrong service/env | UST misconfiguration | Check Unified Service Tag labels on the Deployment | + +Key silent failure modes (SSI produces no error for these): +- Existing ddtrace or OpenTelemetry instrumentation detected: SSI silently disables itself +- Unsupported runtime version: silently skipped +- `admission.datadoghq.com/enabled: "false"` annotation on pod: webhook skips the pod +- Pod not restarted after SSI enabled: injection happens at pod startup only +- Pod in the Datadog Agent namespace: SSI never instruments its own namespace +{{< /agent-only >}} + ## Troubleshooting methods You can troubleshoot injection issues in two ways: by using Fleet Automation in Datadog or by manually verifying at the container level. @@ -387,3 +437,5 @@ Collect the following details if troubleshooting injection in a Kubernetes envir [10]: /tracing/trace_collection/dd_libraries/dotnet-core/#installation-and-getting-started [11]: /tracing/guide/injectors/ [12]: /tracing/trace_collection/automatic_instrumentation/single-step-apm/#instrument-sdks-across-applications +[13]: /tracing/trace_collection/single-step-apm/compatibility/ +[14]: /tracing/trace_collection/dd_libraries/nodejs/ From b4d2bb96058c7de17b79597ff3a5713099b0a8d6 Mon Sep 17 00:00:00 2001 From: Brett Blue Date: Wed, 29 Apr 2026 14:07:13 -0600 Subject: [PATCH 02/13] Add strong page descriptions to all pages in the APM onboarding path Per AI readability guidelines, page descriptions in frontmatter are critical for retrieval. Adds specific, action-oriented descriptions to the Set Up APM entry page, SSI index page, and SSI troubleshooting page. --- content/en/tracing/trace_collection/_index.md | 2 +- content/en/tracing/trace_collection/single-step-apm/_index.md | 1 + .../tracing/trace_collection/single-step-apm/troubleshooting.md | 1 + 3 files changed, 3 insertions(+), 1 deletion(-) diff --git a/content/en/tracing/trace_collection/_index.md b/content/en/tracing/trace_collection/_index.md index 237743d6a2f..3b7b6342886 100644 --- a/content/en/tracing/trace_collection/_index.md +++ b/content/en/tracing/trace_collection/_index.md @@ -1,6 +1,6 @@ --- title: Set Up APM -description: "Get Started with Datadog APM" +description: "Set up Datadog APM to collect traces from your applications. Choose Single Step Instrumentation for automatic setup on Kubernetes, Linux, Docker, or Windows, or use manually managed SDKs for full configuration control." aliases: - /tracing/setup - /tracing/send_traces/ diff --git a/content/en/tracing/trace_collection/single-step-apm/_index.md b/content/en/tracing/trace_collection/single-step-apm/_index.md index aa240ee84ee..e98e5195608 100644 --- a/content/en/tracing/trace_collection/single-step-apm/_index.md +++ b/content/en/tracing/trace_collection/single-step-apm/_index.md @@ -1,5 +1,6 @@ --- title: Single Step APM Instrumentation +description: "Automatically instrument applications with Datadog APM using Single Step Instrumentation (SSI). Install the Datadog Agent and enable tracing with no code changes on Kubernetes, Linux, Docker, or Windows." aliases: - /tracing/trace_collection/admission_controller/ - /tracing/trace_collection/library_injection_local/ diff --git a/content/en/tracing/trace_collection/single-step-apm/troubleshooting.md b/content/en/tracing/trace_collection/single-step-apm/troubleshooting.md index d5182399144..24ebfc75390 100644 --- a/content/en/tracing/trace_collection/single-step-apm/troubleshooting.md +++ b/content/en/tracing/trace_collection/single-step-apm/troubleshooting.md @@ -1,5 +1,6 @@ --- title: Troubleshooting Single Step APM +description: "Diagnose and fix Single Step Instrumentation (SSI) issues on Kubernetes, Linux, Docker, and Windows. Covers missing traces, silent failure modes, injection verification, Fleet Automation diagnostics, and language-specific troubleshooting." aliases: - /tracing/trace_collection/automatic_instrumentation/single-step-apm/troubleshooting/ further_reading: From 3b5ae9de41dc936cdf5a83fe2c91215b17322abb Mon Sep 17 00:00:00 2001 From: Brett Blue Date: Wed, 29 Apr 2026 14:16:56 -0600 Subject: [PATCH 03/13] Address review findings from PR #36370 Fix broken link to dynamic instrumentation (use canonical path). Fix label vs annotation mismatch in troubleshooting triage. Add build.gradle to troubleshooting grep command. Fix agent-only tab reference to name all three tabs. Add opt-out label check to K8s agent-only diagnostic block. Add shell language tags to grep code blocks. Expand collapse titles to use full Unified Service Tags name. Fix Datadog app vale warning. --- .../trace_collection/single-step-apm/kubernetes.md | 14 +++++++------- .../single-step-apm/troubleshooting.md | 8 ++++---- 2 files changed, 11 insertions(+), 11 deletions(-) diff --git a/content/en/tracing/trace_collection/single-step-apm/kubernetes.md b/content/en/tracing/trace_collection/single-step-apm/kubernetes.md index c7ce234f7e4..c5163416810 100644 --- a/content/en/tracing/trace_collection/single-step-apm/kubernetes.md +++ b/content/en/tracing/trace_collection/single-step-apm/kubernetes.md @@ -32,7 +32,7 @@ Single Step Instrumentation (SSI) installs the Datadog Agent and instruments you - [kubectl][2] for installing the Datadog Agent - A supported language runtime per the [SSI compatibility guide][36] - **No existing tracer dependencies.** SSI silently disables itself if it detects `ddtrace`, `dd-trace`, an OpenTelemetry SDK, or `-javaagent` in your application. Before enabling SSI, check your dependency manifests: - ``` + ```shell grep -rn "ddtrace\|dd-trace\|opentelemetry" requirements.txt package.json Gemfile go.mod pom.xml build.gradle 2>/dev/null ``` Remove any matches and rebuild your application image before proceeding. @@ -84,7 +84,7 @@ These steps enable SSI across your entire cluster. To instrument specific namesp {{< tabs >}} {{% tab "In-app wizard" %}} -The Datadog app generates a configuration file with SSI enabled: +Datadog generates a configuration file with SSI enabled: 1. Go to the [Install the Datadog Agent on Kubernetes][11] page. 1. Choose your installation method, select an API key, and set up the Operator or Helm repository. @@ -156,7 +156,7 @@ kubectl rollout restart deployment/ -n
If your cluster enforces PodSecurity restricted policies, the Datadog init container may be blocked with: allowPrivilegeEscalation is false or violates PodSecurity "restricted:latest". See the SSI troubleshooting guide for the required security context configuration.
{{< agent-only >}} -Use the Helm or Datadog Operator tab above. Add `instrumentation.enabled: true` to the user's existing manifest. Do not create a separate manifest. +Use one of the three tabs (In-app wizard, Helm, or Datadog Operator). For agents helping users, prefer the Helm or Operator tab. Add `instrumentation.enabled: true` to the user's existing manifest. Do not create a separate manifest. If the user wants to pin SDK versions, recommend these defaults: `java: "1"`, `python: "2"`, `js: "5"`, `dotnet: "3"`, `ruby: "2"`, `php: "1"` @@ -223,7 +223,7 @@ After the user restarts their pods, verify with these checks: No init container (injection never happened) — check: - Namespace targeting: `kubectl get datadogagent datadog -n -o yaml | grep -A 15 instrumentation` -- Pod annotations: `kubectl get pod -n -o yaml | grep -A 5 annotations` +- Opt-out label: `kubectl get pod -n -o jsonpath='{.metadata.labels.admission\.datadoghq\.com/enabled}'` — if "false", the Admission Controller skips this pod - Pod labels: `kubectl get pod -n --show-labels` - Admission webhook registered: `kubectl get mutatingwebhookconfigurations | grep datadog` - Cluster Agent running: `kubectl get pods -n -l app=datadog-cluster-agent` @@ -267,7 +267,7 @@ datadog: Requires `datadog-agent` 7.69+, `datadog-operator` 1.16.0+, or `datadog-helm-chart` 3.120.0+. -{{% collapse-content title="Configure USTs with ddTraceConfigs" level="h3" expanded=false %}} +{{% collapse-content title="Configure Unified Service Tags with ddTraceConfigs" level="h3" expanded=false %}} For granular control over specific workloads, use `ddTraceConfigs` to map labels to service configurations: @@ -297,7 +297,7 @@ datadog: {{% /collapse-content %}} -{{% collapse-content title="Configure USTs in deployment manifests" level="h3" expanded=false %}} +{{% collapse-content title="Configure Unified Service Tags in deployment manifests" level="h3" expanded=false %}} If your labels are not suitable for automatic extraction, set USTs directly in your deployment manifests with environment variables. This requires modifying each deployment individually. @@ -313,7 +313,7 @@ After traces are flowing: {{< nextlink href="/tracing/trace_explorer/" >}}Trace Explorer: search and analyze your traces{{< /nextlink >}} {{< nextlink href="/tracing/services/service_page/" >}}Service Page: monitor service health and performance{{< /nextlink >}} {{< nextlink href="/tracing/trace_collection/custom_instrumentation/" >}}Custom instrumentation: add application-specific spans{{< /nextlink >}} - {{< nextlink href="/dynamic_instrumentation/" >}}Dynamic Instrumentation: add custom spans without redeploying{{< /nextlink >}} + {{< nextlink href="/tracing/trace_collection/dynamic_instrumentation/" >}}Dynamic Instrumentation: add custom spans without redeploying{{< /nextlink >}} {{< /whatsnext >}} ## Advanced SSI configuration diff --git a/content/en/tracing/trace_collection/single-step-apm/troubleshooting.md b/content/en/tracing/trace_collection/single-step-apm/troubleshooting.md index 24ebfc75390..1cb45c13fc8 100644 --- a/content/en/tracing/trace_collection/single-step-apm/troubleshooting.md +++ b/content/en/tracing/trace_collection/single-step-apm/troubleshooting.md @@ -23,14 +23,14 @@ If you enabled SSI but don't see traces, work through these checks in order: 1. **Did you restart your pods or processes after enabling SSI?** SSI injects at startup. Existing pods and processes are not instrumented until restarted. 2. **Does your application have existing tracer dependencies?** SSI silently disables itself if it detects `ddtrace`, `dd-trace`, OpenTelemetry SDK, or `-javaagent` in your application. Check your dependency manifests: - ``` - grep -rn "ddtrace\|dd-trace\|opentelemetry" requirements.txt package.json Gemfile go.mod pom.xml 2>/dev/null + ```shell + grep -rn "ddtrace\|dd-trace\|opentelemetry" requirements.txt package.json Gemfile go.mod pom.xml build.gradle 2>/dev/null ``` Remove these dependencies and rebuild your application if found. 3. **Is your application in the same namespace as the Datadog Agent?** SSI does not instrument pods in the Agent namespace. -4. **Is a namespace or pod selector filtering your application out?** Check your SSI configuration for `enabledNamespaces`, `disabledNamespaces`, or `podSelector` targets that may not match your application's namespace or labels. Also check for the `admission.datadoghq.com/enabled: "false"` annotation on the pod, which tells the Admission Controller to skip it. +4. **Is a namespace or pod selector filtering your application out?** Check your SSI configuration for `enabledNamespaces`, `disabledNamespaces`, or `podSelector` targets that may not match your application's namespace or labels. Also check for the `admission.datadoghq.com/enabled: "false"` label on the pod, which tells the Admission Controller to skip it. 5. **Is the runtime version supported?** Check the [SSI compatibility guide][13]. @@ -61,7 +61,7 @@ Diagnosis by signal: Key silent failure modes (SSI produces no error for these): - Existing ddtrace or OpenTelemetry instrumentation detected: SSI silently disables itself - Unsupported runtime version: silently skipped -- `admission.datadoghq.com/enabled: "false"` annotation on pod: webhook skips the pod +- `admission.datadoghq.com/enabled: "false"` label on pod: webhook skips the pod - Pod not restarted after SSI enabled: injection happens at pod startup only - Pod in the Datadog Agent namespace: SSI never instruments its own namespace {{< /agent-only >}} From 1372684b1dfd6af43d070d94f19bc0f4958349e7 Mon Sep 17 00:00:00 2001 From: Brett Blue Date: Thu, 30 Apr 2026 16:42:13 -0600 Subject: [PATCH 04/13] Expand Kubernetes SSI setup with bootstrap steps and restart warnings Add full install-from-scratch steps (Helm repo, namespace, secret, site/clusterName) to the Helm and Operator tabs so new users don't need to cross-reference other pages. Add outage warnings before pod restart commands. Clarify Alpine/musl support nuance (Ruby requires glibc). Improve agent-only guidance to check existing Agent state before modifying config. --- .../single-step-apm/kubernetes.md | 89 ++++++++++++++++--- 1 file changed, 76 insertions(+), 13 deletions(-) diff --git a/content/en/tracing/trace_collection/single-step-apm/kubernetes.md b/content/en/tracing/trace_collection/single-step-apm/kubernetes.md index c5163416810..907c98fd34d 100644 --- a/content/en/tracing/trace_collection/single-step-apm/kubernetes.md +++ b/content/en/tracing/trace_collection/single-step-apm/kubernetes.md @@ -37,7 +37,7 @@ Single Step Instrumentation (SSI) installs the Datadog Agent and instruments you ``` Remove any matches and rebuild your application image before proceeding. - **Node.js: CommonJS only.** SSI does not support ECMAScript Modules (ESM). If your application uses `import` syntax or sets `"type": "module"` in `package.json`, use [manually managed SDKs][44] instead. -- **Alpine and musl-based images are supported.** Kubernetes SSI injects through the `LD_PRELOAD` environment variable, not `/etc/ld.so.preload`, so musl libc images work without issues. +- **Alpine and musl-based images:** Kubernetes SSI injects through the `LD_PRELOAD` environment variable, not `/etc/ld.so.preload`, so the Kubernetes injector supports musl-based images. Language and runtime compatibility still apply; for example, Ruby SSI requires glibc and is not compatible with Alpine or other musl-based images. {{< agent-only >}} Resolve these variables before starting: @@ -93,7 +93,10 @@ Datadog generates a configuration file with SSI enabled: {{< img src="tracing/trace_collection/k8s-apm-instrumentation-toggle.jpg" alt="APM Instrumentation toggle in the Kubernetes Agent installation wizard" style="width:100%;" >}} 1. Deploy the Agent with the generated configuration file. -1. Restart your application pods: +1. Coordinate a restart of your application pods. + +
Restarting application pods can cause a brief outage. Coordinate the restart with the application owner before running this command.
+ ```shell kubectl rollout restart deployment/ -n ``` @@ -103,22 +106,42 @@ Datadog generates a configuration file with SSI enabled: {{% /tab %}} {{% tab "Helm" %}} -Add the following to your `datadog-values.yaml`: +If the Datadog Agent is not installed, set `DD_API_KEY` to your [Datadog API key][45]. Then, add the Datadog Helm repository and create a Kubernetes Secret: + +```shell +helm repo add datadog https://helm.datadoghq.com +helm repo update +kubectl create namespace datadog --dry-run=client -o yaml | kubectl apply -f - +kubectl create secret generic datadog-secret --from-literal api-key=$DD_API_KEY -n datadog --dry-run=client -o yaml | kubectl apply -f - +``` + +If you install the Agent in a different namespace, replace `datadog` with your Agent namespace. + +Create or update `datadog-values.yaml`: ```yaml datadog: + apiKeyExistingSecret: datadog-secret + clusterName: + site: apm: instrumentation: enabled: true ``` -Deploy the Agent: +Replace `` with your Kubernetes cluster name and `` with your [Datadog site][46]. + +If the Agent is already installed, add `apm.instrumentation.enabled: true` to your existing `datadog-values.yaml` and keep your existing API key, site, and cluster name configuration. + +Deploy or update the Agent: ```shell -helm upgrade datadog-agent -f datadog-values.yaml datadog/datadog +helm upgrade --install datadog-agent -f datadog-values.yaml datadog/datadog -n datadog ``` -Restart your application pods: +Coordinate a restart of your application pods: + +
Restarting application pods can cause a brief outage. Coordinate the restart with the application owner before running this command.
```shell kubectl rollout restart deployment/ -n @@ -127,22 +150,52 @@ kubectl rollout restart deployment/ -n {{% /tab %}} {{% tab "Datadog Operator" %}} -Add the following to your `datadog-agent.yaml`: +If the Datadog Operator and Agent are not installed, set `DD_API_KEY` to your [Datadog API key][45]. Then, install the Operator and create a Kubernetes Secret: + +```shell +helm repo add datadog https://helm.datadoghq.com +helm repo update +helm upgrade --install datadog-operator datadog/datadog-operator --namespace datadog --create-namespace +kubectl create secret generic datadog-secret --from-literal api-key=$DD_API_KEY -n datadog --dry-run=client -o yaml | kubectl apply -f - +``` + +If you install the Agent in a different namespace, replace `datadog` with your Agent namespace. + +Create or update `datadog-agent.yaml`: ```yaml -features: - apm: - instrumentation: - enabled: true +apiVersion: datadoghq.com/v2alpha1 +kind: DatadogAgent +metadata: + name: datadog + namespace: datadog +spec: + global: + clusterName: + site: + credentials: + apiSecret: + secretName: datadog-secret + keyName: api-key + features: + apm: + instrumentation: + enabled: true ``` +Replace `` with your Kubernetes cluster name and `` with your [Datadog site][46]. + +If the Agent is already installed with the Datadog Operator, add `features.apm.instrumentation.enabled: true` to your existing `DatadogAgent` manifest and keep your existing site, credentials, and cluster name configuration. + Apply the manifest: ```shell kubectl apply -f datadog-agent.yaml ``` -Restart your application pods: +Coordinate a restart of your application pods: + +
Restarting application pods can cause a brief outage. Coordinate the restart with the application owner before running this command.
```shell kubectl rollout restart deployment/ -n @@ -156,7 +209,15 @@ kubectl rollout restart deployment/ -n
If your cluster enforces PodSecurity restricted policies, the Datadog init container may be blocked with: allowPrivilegeEscalation is false or violates PodSecurity "restricted:latest". See the SSI troubleshooting guide for the required security context configuration.
{{< agent-only >}} -Use one of the three tabs (In-app wizard, Helm, or Datadog Operator). For agents helping users, prefer the Helm or Operator tab. Add `instrumentation.enabled: true` to the user's existing manifest. Do not create a separate manifest. +Use one of the three tabs (In-app wizard, Helm, or Datadog Operator). For agents helping users, prefer the Helm or Operator tab. + +Before changing configuration, check whether the Datadog Agent is already installed: +- `helm list -A | grep -i datadog` +- `kubectl get datadogagent -A` + +If the Agent is not installed, follow the bootstrap steps in the selected tab: add the Datadog Helm repository, create the `datadog` namespace, create `datadog-secret`, set `site` and `clusterName`, and deploy the Agent. + +If the Agent is already installed, add `instrumentation.enabled: true` to the user's existing Helm values or `DatadogAgent` manifest. Do not create a separate manifest. If the user wants to pin SDK versions, recommend these defaults: `java: "1"`, `python: "2"`, `js: "5"`, `dotnet: "3"`, `ruby: "2"`, `php: "1"` @@ -790,3 +851,5 @@ If you encounter problems with SSI, see the [SSI troubleshooting guide][35]. [42]: https://app.datadoghq.com/apm/services [43]: https://app.datadoghq.com/apm/traces [44]: /tracing/trace_collection/dd_libraries/nodejs/ +[45]: https://app.datadoghq.com/organization-settings/api-keys +[46]: /getting_started/site/ From 148d43cd4caefd8cb876b26d9160511f7da38bcb Mon Sep 17 00:00:00 2001 From: Brett Blue Date: Thu, 30 Apr 2026 16:50:58 -0600 Subject: [PATCH 05/13] Fix agent-only routing order and resolve NODE_HOSTNAME variable Ask for language before platform in the APM setup routing block so unsupported SSI languages (Go, C++, Rust) bail early. Add a node discovery step before the troubleshooting triage commands so NODE_HOSTNAME is resolved before it is used. --- content/en/tracing/trace_collection/_index.md | 6 +++--- .../trace_collection/single-step-apm/troubleshooting.md | 3 +++ 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/content/en/tracing/trace_collection/_index.md b/content/en/tracing/trace_collection/_index.md index 3b7b6342886..8bc0d43ba79 100644 --- a/content/en/tracing/trace_collection/_index.md +++ b/content/en/tracing/trace_collection/_index.md @@ -72,13 +72,13 @@ After you have APM running, you can add custom spans for visibility into applica {{< agent-only >}} If a user asks to set up APM, tracing, or distributed tracing, start with Single Step Instrumentation (SSI) unless they specifically request manual SDK control. -Ask what platform they're running on: +First, ask what language they're using. If the language is Go, C++, or Rust, SSI is not available — direct them to manually managed SDKs at /tracing/trace_collection/dd_libraries/. + +Then ask what platform they're running on: - Kubernetes → /tracing/trace_collection/single-step-apm/kubernetes - Linux host or VM → /tracing/trace_collection/single-step-apm/linux - Docker → /tracing/trace_collection/single-step-apm/docker - Windows IIS → /tracing/trace_collection/single-step-apm/windows - -If their language is Go, C++, or Rust, SSI is not available — direct them to manually managed SDKs at /tracing/trace_collection/dd_libraries/. {{< /agent-only >}} ## Further reading diff --git a/content/en/tracing/trace_collection/single-step-apm/troubleshooting.md b/content/en/tracing/trace_collection/single-step-apm/troubleshooting.md index 1cb45c13fc8..cf0c3485c44 100644 --- a/content/en/tracing/trace_collection/single-step-apm/troubleshooting.md +++ b/content/en/tracing/trace_collection/single-step-apm/troubleshooting.md @@ -41,6 +41,9 @@ If none of these apply, continue with the detailed troubleshooting methods below {{< agent-only >}} When diagnosing missing traces after SSI enablement, run these diagnostic commands: +First, resolve the node where the pod is running: +- `kubectl get pod -n -o jsonpath='{.spec.nodeName}'` → use this value as `NODE_HOSTNAME` below + Triage (run all simultaneously): - `kubectl get pod -n -o jsonpath='{.spec.initContainers[*].name}'` - `kubectl describe pod -n | grep -A 10 "Events:"` From 5ab1640141b61c3bc84bfc71292d5206c417ee0b Mon Sep 17 00:00:00 2001 From: Brett Blue Date: Thu, 30 Apr 2026 17:02:43 -0600 Subject: [PATCH 06/13] Clean up prerequisites and tab formatting consistency Rewrite Alpine/musl bullet as a constraint ("Ruby: glibc only") to match the pattern of neighboring bullets. Move "already installed" shortcut before the full YAML blocks in Helm and Operator tabs so existing users can skip early. Standardize restart warning text across all three tabs. --- .../single-step-apm/kubernetes.md | 26 +++++++++---------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/content/en/tracing/trace_collection/single-step-apm/kubernetes.md b/content/en/tracing/trace_collection/single-step-apm/kubernetes.md index 907c98fd34d..eca046388e7 100644 --- a/content/en/tracing/trace_collection/single-step-apm/kubernetes.md +++ b/content/en/tracing/trace_collection/single-step-apm/kubernetes.md @@ -37,7 +37,7 @@ Single Step Instrumentation (SSI) installs the Datadog Agent and instruments you ``` Remove any matches and rebuild your application image before proceeding. - **Node.js: CommonJS only.** SSI does not support ECMAScript Modules (ESM). If your application uses `import` syntax or sets `"type": "module"` in `package.json`, use [manually managed SDKs][44] instead. -- **Alpine and musl-based images:** Kubernetes SSI injects through the `LD_PRELOAD` environment variable, not `/etc/ld.so.preload`, so the Kubernetes injector supports musl-based images. Language and runtime compatibility still apply; for example, Ruby SSI requires glibc and is not compatible with Alpine or other musl-based images. +- **Ruby: glibc only.** Ruby SSI requires glibc and is not compatible with Alpine or other musl-based images. Other SSI-supported languages work on musl-based images. {{< agent-only >}} Resolve these variables before starting: @@ -93,9 +93,9 @@ Datadog generates a configuration file with SSI enabled: {{< img src="tracing/trace_collection/k8s-apm-instrumentation-toggle.jpg" alt="APM Instrumentation toggle in the Kubernetes Agent installation wizard" style="width:100%;" >}} 1. Deploy the Agent with the generated configuration file. -1. Coordinate a restart of your application pods. +1. Restart your application pods. -
Restarting application pods can cause a brief outage. Coordinate the restart with the application owner before running this command.
+
Restarting pods can cause a brief service interruption. Coordinate with the application owner before proceeding.
```shell kubectl rollout restart deployment/ -n @@ -117,7 +117,9 @@ kubectl create secret generic datadog-secret --from-literal api-key=$DD_API_KEY If you install the Agent in a different namespace, replace `datadog` with your Agent namespace. -Create or update `datadog-values.yaml`: +If the Agent is already installed, add `apm.instrumentation.enabled: true` to your existing `datadog-values.yaml` and skip to the `helm upgrade` step below. + +Create a `datadog-values.yaml`: ```yaml datadog: @@ -131,17 +133,15 @@ datadog: Replace `` with your Kubernetes cluster name and `` with your [Datadog site][46]. -If the Agent is already installed, add `apm.instrumentation.enabled: true` to your existing `datadog-values.yaml` and keep your existing API key, site, and cluster name configuration. - Deploy or update the Agent: ```shell helm upgrade --install datadog-agent -f datadog-values.yaml datadog/datadog -n datadog ``` -Coordinate a restart of your application pods: +Restart your application pods: -
Restarting application pods can cause a brief outage. Coordinate the restart with the application owner before running this command.
+
Restarting pods can cause a brief service interruption. Coordinate with the application owner before proceeding.
```shell kubectl rollout restart deployment/ -n @@ -161,7 +161,9 @@ kubectl create secret generic datadog-secret --from-literal api-key=$DD_API_KEY If you install the Agent in a different namespace, replace `datadog` with your Agent namespace. -Create or update `datadog-agent.yaml`: +If the Agent is already installed with the Datadog Operator, add `features.apm.instrumentation.enabled: true` to your existing `DatadogAgent` manifest and skip to the `kubectl apply` step below. + +Create a `datadog-agent.yaml`: ```yaml apiVersion: datadoghq.com/v2alpha1 @@ -185,17 +187,15 @@ spec: Replace `` with your Kubernetes cluster name and `` with your [Datadog site][46]. -If the Agent is already installed with the Datadog Operator, add `features.apm.instrumentation.enabled: true` to your existing `DatadogAgent` manifest and keep your existing site, credentials, and cluster name configuration. - Apply the manifest: ```shell kubectl apply -f datadog-agent.yaml ``` -Coordinate a restart of your application pods: +Restart your application pods: -
Restarting application pods can cause a brief outage. Coordinate the restart with the application owner before running this command.
+
Restarting pods can cause a brief service interruption. Coordinate with the application owner before proceeding.
```shell kubectl rollout restart deployment/ -n From f1f7f441723d351edee553cfea38e04dc1ac2c1e Mon Sep 17 00:00:00 2001 From: Brett Blue Date: Wed, 6 May 2026 15:22:56 -0600 Subject: [PATCH 07/13] Address codex review findings MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 1. Add platform-specific triage sections (Kubernetes, Linux) to troubleshooting — was K8s-only despite page covering all platforms. Added Linux checks: /etc/ld.so.preload, musl/glibc, SELinux/AppArmor. 2. Add dd-java-agent and javaagent to tracer grep commands on K8s and troubleshooting pages. Add note about checking Dockerfiles and JAVA_TOOL_OPTIONS for Java. 3. Fix stale python: "2" in agent-only SDK version defaults — should be python: "3" to match current docs and best-practice examples. 4. Update footnote and further_reading links from old alias paths (/automatic_instrumentation/single-step-apm/) to canonical paths (/single-step-apm/) on SSI index and troubleshooting pages. Preserves aliases in frontmatter for redirects. --- .../single-step-apm/_index.md | 10 ++--- .../single-step-apm/kubernetes.md | 8 ++-- .../single-step-apm/troubleshooting.md | 44 +++++++++++++------ 3 files changed, 39 insertions(+), 23 deletions(-) diff --git a/content/en/tracing/trace_collection/single-step-apm/_index.md b/content/en/tracing/trace_collection/single-step-apm/_index.md index e98e5195608..ce72324c642 100644 --- a/content/en/tracing/trace_collection/single-step-apm/_index.md +++ b/content/en/tracing/trace_collection/single-step-apm/_index.md @@ -12,7 +12,7 @@ further_reading: - link: /tracing/guide/injectors tag: Documentation text: Understanding injector behavior with Single Step Instrumentation - - link: /tracing/trace_collection/automatic_instrumentation/single-step-apm/troubleshooting/ + - link: /tracing/trace_collection/single-step-apm/troubleshooting/ tag: Documentation text: "Troubleshooting Single Step APM" - link: https://learn.datadoghq.com/courses/troubleshooting-apm-instrumentation-on-a-host @@ -66,16 +66,16 @@ If you encounter problems enabling APM with SSI, see the [SSI troubleshooting gu [3]: /tracing/software_catalog/ [4]: /tracing/glossary/#instrumentation [5]: /containers/cluster_agent/admission_controller/ -[6]: /tracing/trace_collection/automatic_instrumentation/single-step-apm/compatibility +[6]: /tracing/trace_collection/single-step-apm/compatibility [7]: /tracing/trace_collection/custom_instrumentation/ [8]: /tracing/guide/injectors -[9]: /tracing/trace_collection/automatic_instrumentation/single-step-apm/kubernetes/?tab=installingwithdatadogoperator#configure-instrumentation-for-namespaces-and-pods +[9]: /tracing/trace_collection/single-step-apm/kubernetes/?tab=installingwithdatadogoperator#configure-instrumentation-for-namespaces-and-pods [10]: /tracing/trace_collection/library_config/ [11]: /tracing/metrics/runtime_metrics/ [12]: /tracing/software_catalog/ [13]: /tracing/glossary/#instrumentation [14]: /getting_started/tagging/unified_service_tagging -[15]: /tracing/trace_collection/automatic_instrumentation/single-step-apm/troubleshooting +[15]: /tracing/trace_collection/single-step-apm/troubleshooting [16]: /tracing/trace_collection/custom_instrumentation/ [17]: /tracing/trace_collection/library_config/application_monitoring_yaml/ -[18]: /tracing/trace_collection/automatic_instrumentation/single-step-apm/compatibility/ +[18]: /tracing/trace_collection/single-step-apm/compatibility/ diff --git a/content/en/tracing/trace_collection/single-step-apm/kubernetes.md b/content/en/tracing/trace_collection/single-step-apm/kubernetes.md index eca046388e7..0f30f1484a6 100644 --- a/content/en/tracing/trace_collection/single-step-apm/kubernetes.md +++ b/content/en/tracing/trace_collection/single-step-apm/kubernetes.md @@ -31,11 +31,11 @@ Single Step Instrumentation (SSI) installs the Datadog Agent and instruments you - [Helm][1] for deploying the Datadog Operator - [kubectl][2] for installing the Datadog Agent - A supported language runtime per the [SSI compatibility guide][36] -- **No existing tracer dependencies.** SSI silently disables itself if it detects `ddtrace`, `dd-trace`, an OpenTelemetry SDK, or `-javaagent` in your application. Before enabling SSI, check your dependency manifests: +- **No existing tracer dependencies.** SSI silently disables itself if it detects `ddtrace`, `dd-trace`, an OpenTelemetry SDK, or `-javaagent` in your application. Before enabling SSI, check your dependency manifests and startup scripts: ```shell - grep -rn "ddtrace\|dd-trace\|opentelemetry" requirements.txt package.json Gemfile go.mod pom.xml build.gradle 2>/dev/null + grep -rn "ddtrace\|dd-trace\|opentelemetry\|dd-java-agent\|javaagent" requirements.txt package.json Gemfile go.mod pom.xml build.gradle 2>/dev/null ``` - Remove any matches and rebuild your application image before proceeding. + For Java, also check Dockerfiles and startup scripts for `-javaagent` flags, and check the `JAVA_TOOL_OPTIONS` environment variable. Remove any matches and rebuild your application image before proceeding. - **Node.js: CommonJS only.** SSI does not support ECMAScript Modules (ESM). If your application uses `import` syntax or sets `"type": "module"` in `package.json`, use [manually managed SDKs][44] instead. - **Ruby: glibc only.** Ruby SSI requires glibc and is not compatible with Alpine or other musl-based images. Other SSI-supported languages work on musl-based images. @@ -220,7 +220,7 @@ If the Agent is not installed, follow the bootstrap steps in the selected tab: a If the Agent is already installed, add `instrumentation.enabled: true` to the user's existing Helm values or `DatadogAgent` manifest. Do not create a separate manifest. If the user wants to pin SDK versions, recommend these defaults: -`java: "1"`, `python: "2"`, `js: "5"`, `dotnet: "3"`, `ruby: "2"`, `php: "1"` +`java: "1"`, `python: "3"`, `js: "5"`, `dotnet: "3"`, `ruby: "2"`, `php: "1"` After applying the config, confirm with the user before restarting their pods. After restart, wait for pods to be ready: `kubectl wait --for=condition=Ready pod -l app= -n --timeout=120s` diff --git a/content/en/tracing/trace_collection/single-step-apm/troubleshooting.md b/content/en/tracing/trace_collection/single-step-apm/troubleshooting.md index cf0c3485c44..071a1f99721 100644 --- a/content/en/tracing/trace_collection/single-step-apm/troubleshooting.md +++ b/content/en/tracing/trace_collection/single-step-apm/troubleshooting.md @@ -4,7 +4,7 @@ description: "Diagnose and fix Single Step Instrumentation (SSI) issues on Kuber aliases: - /tracing/trace_collection/automatic_instrumentation/single-step-apm/troubleshooting/ further_reading: -- link: /tracing/trace_collection/automatic_instrumentation/single-step-apm/ +- link: /tracing/trace_collection/single-step-apm/ tag: Documentation text: Single Step APM Instrumentation - link: https://learn.datadoghq.com/courses/troubleshooting-apm-instrumentation-on-a-host @@ -20,21 +20,37 @@ Single Step Instrumentation (SSI) helps instrument applications by automatically If you enabled SSI but don't see traces, work through these checks in order: -1. **Did you restart your pods or processes after enabling SSI?** SSI injects at startup. Existing pods and processes are not instrumented until restarted. +### All platforms -2. **Does your application have existing tracer dependencies?** SSI silently disables itself if it detects `ddtrace`, `dd-trace`, OpenTelemetry SDK, or `-javaagent` in your application. Check your dependency manifests: +1. **Did you restart your application after enabling SSI?** SSI injects at startup. Existing processes and pods are not instrumented until restarted. + +2. **Does your application have existing tracer dependencies?** SSI silently disables itself if it detects `ddtrace`, `dd-trace`, an OpenTelemetry SDK, or `-javaagent` in your application. Check your dependency manifests and startup scripts: ```shell - grep -rn "ddtrace\|dd-trace\|opentelemetry" requirements.txt package.json Gemfile go.mod pom.xml build.gradle 2>/dev/null + grep -rn "ddtrace\|dd-trace\|opentelemetry\|dd-java-agent\|javaagent" requirements.txt package.json Gemfile go.mod pom.xml build.gradle 2>/dev/null ``` - Remove these dependencies and rebuild your application if found. + For Java, also check Dockerfiles and startup scripts for `-javaagent` flags, and check the `JAVA_TOOL_OPTIONS` environment variable. Remove any matches and rebuild your application before proceeding. + +3. **Is the runtime version supported?** Check the [SSI compatibility guide][13]. + +4. **For Node.js: is your application using ECMAScript Modules (ESM)?** SSI does not support ESM. If your application uses `import` syntax or sets `"type": "module"` in `package.json`, use [manually managed SDKs][14] instead. + +### Kubernetes -3. **Is your application in the same namespace as the Datadog Agent?** SSI does not instrument pods in the Agent namespace. +5. **Is your application in the same namespace as the Datadog Agent?** SSI does not instrument pods in the Agent namespace. -4. **Is a namespace or pod selector filtering your application out?** Check your SSI configuration for `enabledNamespaces`, `disabledNamespaces`, or `podSelector` targets that may not match your application's namespace or labels. Also check for the `admission.datadoghq.com/enabled: "false"` label on the pod, which tells the Admission Controller to skip it. +6. **Is a namespace or pod selector filtering your application out?** Check your SSI configuration for `enabledNamespaces`, `disabledNamespaces`, or `podSelector` targets that may not match your application's namespace or labels. Also check for the `admission.datadoghq.com/enabled: "false"` label on the pod, which tells the Admission Controller to skip it. + +### Linux hosts + +7. **Is `/etc/ld.so.preload` configured?** SSI on Linux uses `/etc/ld.so.preload` to load the injector. If this file is missing or does not contain the Datadog launcher path, SSI is not active. Check: + ```shell + cat /etc/ld.so.preload + ``` + The output should contain `/opt/datadog-packages/datadog-apm-inject/stable/inject/launcher.preload.so`. -5. **Is the runtime version supported?** Check the [SSI compatibility guide][13]. +8. **Is the application using musl libc or a static binary?** The SSI injector on Linux requires glibc. Applications on Alpine or other musl-based distributions, and statically linked binaries (common with Go), cannot be instrumented with SSI. -6. **For Node.js: is your application using ECMAScript Modules (ESM)?** SSI does not support ESM. If your application uses `import` syntax or sets `"type": "module"` in `package.json`, use [manually managed SDKs][14] instead. +9. **Is SELinux or AppArmor blocking the injector?** Security policies can prevent `/etc/ld.so.preload` from being read. Check `getenforce` (SELinux) or `dmesg | grep apparmor` (AppArmor) for denials. If none of these apply, continue with the detailed troubleshooting methods below. @@ -429,10 +445,10 @@ Collect the following details if troubleshooting injection in a Kubernetes envir {{< partial name="whats-next/whats-next.html" >}} -[1]: /tracing/trace_collection/automatic_instrumentation/single-step-apm/kubernetes?tab=agentv764recommended#remove-apm-for-all-services-on-the-infrastructure -[2]: /tracing/trace_collection/automatic_instrumentation/single-step-apm/docker#remove-apm-for-all-services-on-the-infrastructure -[3]: /tracing/trace_collection/automatic_instrumentation/single-step-apm/linux#remove-single-step-apm-instrumentation-from-your-agent -[4]: /tracing/trace_collection/automatic_instrumentation/single-step-apm/windows#remove-single-step-apm-instrumentation-from-your-agent +[1]: /tracing/trace_collection/single-step-apm/kubernetes?tab=agentv764recommended#remove-apm-for-all-services-on-the-infrastructure +[2]: /tracing/trace_collection/single-step-apm/docker#remove-apm-for-all-services-on-the-infrastructure +[3]: /tracing/trace_collection/single-step-apm/linux#remove-single-step-apm-instrumentation-from-your-agent +[4]: /tracing/trace_collection/single-step-apm/windows#remove-single-step-apm-instrumentation-from-your-agent [5]: /containers/guide/sync_container_images/#copy-an-image-to-another-registry-using-crane [6]: https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/#syntax-and-character-set [7]: https://datatracker.ietf.org/doc/html/rfc1035 @@ -440,6 +456,6 @@ Collect the following details if troubleshooting injection in a Kubernetes envir [9]: https://app.datadoghq.com/fleet [10]: /tracing/trace_collection/dd_libraries/dotnet-core/#installation-and-getting-started [11]: /tracing/guide/injectors/ -[12]: /tracing/trace_collection/automatic_instrumentation/single-step-apm/#instrument-sdks-across-applications +[12]: /tracing/trace_collection/single-step-apm/#instrument-sdks-across-applications [13]: /tracing/trace_collection/single-step-apm/compatibility/ [14]: /tracing/trace_collection/dd_libraries/nodejs/ From 6e5267889c2fdc407c4f2b7b8d3827ca050cf6ea Mon Sep 17 00:00:00 2001 From: Brett Blue Date: Wed, 6 May 2026 15:29:10 -0600 Subject: [PATCH 08/13] Clean up prerequisites formatting on K8s SSI page Split prerequisites into three clear sections: requirements (bullet list), tracer dependency check (own heading with code block), and compatibility notes (clean bullet list). Removes inconsistent bold- header-paragraph formatting mixed with list items. --- .../single-step-apm/kubernetes.md | 25 +++++++++++++------ 1 file changed, 18 insertions(+), 7 deletions(-) diff --git a/content/en/tracing/trace_collection/single-step-apm/kubernetes.md b/content/en/tracing/trace_collection/single-step-apm/kubernetes.md index 0f30f1484a6..79cfa897ea7 100644 --- a/content/en/tracing/trace_collection/single-step-apm/kubernetes.md +++ b/content/en/tracing/trace_collection/single-step-apm/kubernetes.md @@ -31,13 +31,24 @@ Single Step Instrumentation (SSI) installs the Datadog Agent and instruments you - [Helm][1] for deploying the Datadog Operator - [kubectl][2] for installing the Datadog Agent - A supported language runtime per the [SSI compatibility guide][36] -- **No existing tracer dependencies.** SSI silently disables itself if it detects `ddtrace`, `dd-trace`, an OpenTelemetry SDK, or `-javaagent` in your application. Before enabling SSI, check your dependency manifests and startup scripts: - ```shell - grep -rn "ddtrace\|dd-trace\|opentelemetry\|dd-java-agent\|javaagent" requirements.txt package.json Gemfile go.mod pom.xml build.gradle 2>/dev/null - ``` - For Java, also check Dockerfiles and startup scripts for `-javaagent` flags, and check the `JAVA_TOOL_OPTIONS` environment variable. Remove any matches and rebuild your application image before proceeding. -- **Node.js: CommonJS only.** SSI does not support ECMAScript Modules (ESM). If your application uses `import` syntax or sets `"type": "module"` in `package.json`, use [manually managed SDKs][44] instead. -- **Ruby: glibc only.** Ruby SSI requires glibc and is not compatible with Alpine or other musl-based images. Other SSI-supported languages work on musl-based images. + +### Check for existing tracer dependencies + +SSI silently disables itself if it detects an existing tracer in your application. Before enabling SSI, check your dependency manifests and startup scripts: + +```shell +grep -rn "ddtrace\|dd-trace\|opentelemetry\|dd-java-agent\|javaagent" requirements.txt package.json Gemfile go.mod pom.xml build.gradle 2>/dev/null +``` + +For Java, also check Dockerfiles and startup scripts for `-javaagent` flags, and check the `JAVA_TOOL_OPTIONS` environment variable. + +If any matches are found, remove the tracer dependencies and rebuild your application image before proceeding. + +### Compatibility notes + +- **Node.js**: SSI requires CommonJS. If your application uses `import` syntax or sets `"type": "module"` in `package.json`, use [manually managed SDKs][44] instead. +- **Ruby**: SSI requires glibc and is not compatible with Alpine or other musl-based images. +- **All other languages**: Alpine and musl-based images are supported on Kubernetes. SSI injects through the `LD_PRELOAD` environment variable, not `/etc/ld.so.preload`. {{< agent-only >}} Resolve these variables before starting: From cd64275280319f7aa793656a08acd643de7cd365 Mon Sep 17 00:00:00 2001 From: Brett Blue Date: Wed, 6 May 2026 15:33:39 -0600 Subject: [PATCH 09/13] Add platform branches to troubleshooting agent-only block The agent-only triage block assumed Kubernetes only. Split into platform-labeled sections: all-platforms pup commands, Kubernetes kubectl diagnostics with signal table, and Linux host diagnostics (ld.so.preload, /proc/maps, libc check, SELinux/AppArmor). Silent failure modes now labeled by platform. --- .../single-step-apm/troubleshooting.md | 41 ++++++++++++------- 1 file changed, 27 insertions(+), 14 deletions(-) diff --git a/content/en/tracing/trace_collection/single-step-apm/troubleshooting.md b/content/en/tracing/trace_collection/single-step-apm/troubleshooting.md index 071a1f99721..5b6f7ae3e24 100644 --- a/content/en/tracing/trace_collection/single-step-apm/troubleshooting.md +++ b/content/en/tracing/trace_collection/single-step-apm/troubleshooting.md @@ -55,34 +55,47 @@ If you enabled SSI but don't see traces, work through these checks in order: If none of these apply, continue with the detailed troubleshooting methods below. {{< agent-only >}} -When diagnosing missing traces after SSI enablement, run these diagnostic commands: +All platforms — start with pup if available: +- `pup traces search --query "service:" --from 1h --limit 5` +- `pup apm services list --env ` +- `pup apm troubleshooting list --hostname --timeframe 1h` -First, resolve the node where the pod is running: -- `kubectl get pod -n -o jsonpath='{.spec.nodeName}'` → use this value as `NODE_HOSTNAME` below +Kubernetes diagnostics: -Triage (run all simultaneously): +Resolve the node hostname first: +- `kubectl get pod -n -o jsonpath='{.spec.nodeName}'` → use as NODE_HOSTNAME below + +Triage (run simultaneously): - `kubectl get pod -n -o jsonpath='{.spec.initContainers[*].name}'` - `kubectl describe pod -n | grep -A 10 "Events:"` - `kubectl get datadogagent datadog -n -o yaml | grep -A 15 instrumentation` -- If pup CLI is available: - - `pup traces search --query "service:" --from 1h --limit 5` - - `pup fleet instrumented-pods list ` - - `pup apm troubleshooting list --hostname --timeframe 1h` +- `pup fleet instrumented-pods list ` -Diagnosis by signal: +Kubernetes diagnosis by signal: | Signal | Likely cause | Next step | |--------|-------------|-----------| -| No init container on pod | Injection never happened | Check namespace targeting, pod-selector, opt-out annotation, webhook registration, pod not restarted | +| No init container on pod | Injection never happened | Check namespace targeting, pod-selector, opt-out label, webhook registration, pod not restarted | | Init container present, no traces | Injection attempted but tracer not reporting | Check for existing ddtrace/OTel dependencies, runtime version, Agent connectivity | | Pod events show CrashLoopBackOff | Init container failure | Check existing instrumentation conflicts, runtime version | | Traces arriving but wrong service/env | UST misconfiguration | Check Unified Service Tag labels on the Deployment | +Linux host diagnostics: +- Check injector is armed: `cat /etc/ld.so.preload` (should contain launcher path) +- Check tracer loaded in process: `grep datadog /proc//maps` +- Check process started after SSI: compare `ps -o lstart -p ` against `/etc/ld.so.preload` modification time +- Check libc type: `ldd --version` (musl = incompatible) +- Check binary type: `file /proc//exe` (statically linked = incompatible) +- Check Agent APM receiver: `datadog-agent status | grep -A 5 "APM Agent"` +- Check security: `getenforce` (SELinux), `dmesg | grep apparmor` (AppArmor) + Key silent failure modes (SSI produces no error for these): -- Existing ddtrace or OpenTelemetry instrumentation detected: SSI silently disables itself +- Existing ddtrace or OpenTelemetry instrumentation: SSI silently disables itself - Unsupported runtime version: silently skipped -- `admission.datadoghq.com/enabled: "false"` label on pod: webhook skips the pod -- Pod not restarted after SSI enabled: injection happens at pod startup only -- Pod in the Datadog Agent namespace: SSI never instruments its own namespace +- Kubernetes: `admission.datadoghq.com/enabled: "false"` label on pod: webhook skips the pod +- Application not restarted after SSI enabled: injection happens at startup only +- Kubernetes: pod in the Datadog Agent namespace: SSI skips its own namespace +- Linux: musl libc or static binary: injector requires glibc +- Linux: SELinux/AppArmor blocking `/etc/ld.so.preload` reads {{< /agent-only >}} ## Troubleshooting methods From 9b773c4436b0b64744a3a2dff7fc78c3531906a2 Mon Sep 17 00:00:00 2001 From: Brett Blue Date: Fri, 17 Jul 2026 14:48:22 -0600 Subject: [PATCH 10/13] Rewrite SSI pages for agentic onboarding: real enable commands, crisp setup, promoted diagnostics --- .../single-step-apm/_index.md | 31 +- .../single-step-apm/compatibility.md | 4 +- .../single-step-apm/docker.md | 97 ++- .../single-step-apm/kubernetes.md | 720 ++++++------------ .../trace_collection/single-step-apm/linux.md | 171 +++-- .../single-step-apm/troubleshooting.md | 316 +++++--- .../single-step-apm/windows.md | 134 ++-- 7 files changed, 658 insertions(+), 815 deletions(-) diff --git a/content/en/tracing/trace_collection/single-step-apm/_index.md b/content/en/tracing/trace_collection/single-step-apm/_index.md index bba20fa91f5..b61f9b8c107 100644 --- a/content/en/tracing/trace_collection/single-step-apm/_index.md +++ b/content/en/tracing/trace_collection/single-step-apm/_index.md @@ -30,21 +30,21 @@ further_reading: --- ## Overview -Single Step Instrumentation (SSI) automatically installs the Datadog SDKs with no additional configuration required, reducing onboarding time from days to minutes. +Single Step Instrumentation (SSI) automatically installs the Datadog SDKs into your applications with no code changes or additional configuration required. -To learn more about how it works, see the [injector guide for Single Step Instrumentation][8]. +To learn more about how it works, see the [injector guide for Single Step Instrumentation][2]. ## Prerequisites 1. Remove any custom instrumentation code from your application and restart it. SSI is automatically disabled if custom instrumentation is detected. -1. Confirm environment compatibility by reviewing the [SSI compatibility guide][18] for supported languages, operating systems, and architectures. +1. Confirm environment compatibility by reviewing the [SSI compatibility guide][5] for supported languages, operating systems, and architectures. ## Instrument SDKs across applications When you [install or update the Datadog Agent][1] with {{< ui >}}APM Instrumentation{{< /ui >}} enabled, the Agent instruments your applications by loading the Datadog SDK into supported processes. This enables distributed tracing by capturing and sending trace data from your services without requiring code changes. After instrumentation, you can optionally: -- [configure Unified Service Tags (USTs)][14] +- [configure Unified Service Tags (USTs)][3] - enable additional SDK-dependent products and features, such as Continuous Profiler or Application Security Monitoring Click on one of the following tiles to learn how to set up SSI for your deployment type: @@ -60,27 +60,14 @@ Click on one of the following tiles to learn how to set up SSI for your deployme ## Troubleshooting -If you encounter problems enabling APM with SSI, see the [SSI troubleshooting guide][15]. +If you encounter problems enabling APM with SSI, see the [SSI troubleshooting guide][4]. ## Further reading {{< partial name="whats-next/whats-next.html" >}} [1]: https://app.datadoghq.com/account/settings/agent/latest -[2]: /tracing/metrics/runtime_metrics/ -[3]: /internal_developer_portal/catalog/ -[4]: /tracing/glossary/#instrumentation -[5]: /containers/cluster_agent/admission_controller/ -[6]: /tracing/trace_collection/single-step-apm/compatibility -[7]: /tracing/trace_collection/custom_instrumentation/ -[8]: /tracing/guide/injectors -[9]: /tracing/trace_collection/single-step-apm/kubernetes/?tab=installingwithdatadogoperator#configure-instrumentation-for-namespaces-and-pods -[10]: /tracing/trace_collection/library_config/ -[11]: /tracing/metrics/runtime_metrics/ -[12]: /internal_developer_portal/catalog/ -[13]: /tracing/glossary/#instrumentation -[14]: /getting_started/tagging/unified_service_tagging -[15]: /tracing/trace_collection/single-step-apm/troubleshooting -[16]: /tracing/trace_collection/custom_instrumentation/ -[17]: /tracing/trace_collection/library_config/application_monitoring_yaml/ -[18]: /tracing/trace_collection/single-step-apm/compatibility/ +[2]: /tracing/guide/injectors +[3]: /getting_started/tagging/unified_service_tagging +[4]: /tracing/trace_collection/single-step-apm/troubleshooting +[5]: /tracing/trace_collection/single-step-apm/compatibility/ diff --git a/content/en/tracing/trace_collection/single-step-apm/compatibility.md b/content/en/tracing/trace_collection/single-step-apm/compatibility.md index 0cfac419d84..ff5111a39d5 100644 --- a/content/en/tracing/trace_collection/single-step-apm/compatibility.md +++ b/content/en/tracing/trace_collection/single-step-apm/compatibility.md @@ -3,7 +3,7 @@ title: Compatibility aliases: - /tracing/trace_collection/automatic_instrumentation/single-step-apm/compatibility/ further_reading: -- link: /tracing/trace_collection/automatic_instrumentation/single-step-apm/ +- link: /tracing/trace_collection/single-step-apm/ tag: "Documentation" text: "Single Step APM Instrumentation" --- @@ -181,7 +181,7 @@ To mitigate: [1]: /tracing/trace_collection/compatibility/python [2]: https://github.com/DataDog/dd-trace-py/releases -[3]: /tracing/trace_collection/automatic_instrumentation/single-step-apm/#instrument-sdks-across-applications +[3]: /tracing/trace_collection/single-step-apm/#instrument-sdks-across-applications [4]: /tracing/trace_collection/dd_libraries/python/ {{< /programming-lang >}} diff --git a/content/en/tracing/trace_collection/single-step-apm/docker.md b/content/en/tracing/trace_collection/single-step-apm/docker.md index efcc360bf30..866fecf7d66 100644 --- a/content/en/tracing/trace_collection/single-step-apm/docker.md +++ b/content/en/tracing/trace_collection/single-step-apm/docker.md @@ -13,54 +13,73 @@ further_reading: ## Overview -In a Docker Linux container, use Single Step Instrumentation (SSI) for APM to install the Datadog Agent and [instrument][14] your applications in one step, with no additional configuration required. +In a Docker Linux container, use Single Step Instrumentation (SSI) for APM to install the Datadog Agent and [instrument][3] your applications in one step, with no additional configuration required. ## Enable APM on your applications -
Before proceeding, confirm that your environment is compatible by reviewing the SSI compatibility guide.
+
Before you begin, confirm that your environment is compatible by reviewing the SSI compatibility guide.
-To enable APM in a Docker Linux container: +{{< tabs >}} +{{% tab "Installation command" %}} -1. In Datadog, go to the [Install the Datadog Agent on Docker][15] page. -1. In the {{< ui >}}Customize my agent install command{{< /ui >}} section, go to {{< ui >}}Additional configuration{{< /ui >}} > {{< ui >}}Application Observability{{< /ui >}}, and turn on {{< ui >}}APM Instrumentation{{< /ui >}}. - - {{< img src="tracing/trace_collection/docker-apm-instrumentation-toggle.png" alt="The 'Customize your agent install command' section of in-app instructions for installing the Datadog Agent on Docker" style="width:100%;" >}} +Run the following command to install the Agent and enable SSI. Replace `` with your [Datadog API key][1] and `` with your [Datadog site][2]. -1. Copy and run the Agent installation command in your Docker container. If the Agent is already running, redeploy the Agent container using the new command. -1. Restart your applications. +```shell +DD_API_KEY= DD_SITE= DD_APM_INSTRUMENTATION_ENABLED=docker \ + bash -c "$(curl -L https://install.datadoghq.com/scripts/install_script_agent7.sh)" +``` + +If the Agent is already running, redeploy the Agent container with the new command. + +{{% /tab %}} +{{% tab "In-app instructions" %}} + +1. In Datadog, go to the [Install the Datadog Agent on Docker][4] page. +1. In the {{< ui >}}Customize my Agent install command{{< /ui >}} section, go to {{< ui >}}Additional configuration{{< /ui >}} > {{< ui >}}Application Observability{{< /ui >}}, and turn on {{< ui >}}APM Instrumentation{{< /ui >}}. + + {{< img src="tracing/trace_collection/docker-apm-instrumentation-toggle.png" alt="The 'Customize your Agent install command' section of in-app instructions for installing the Datadog Agent on Docker" style="width:100%;" >}} + +1. Copy and run the Agent installation command in your Docker container. If the Agent is already running, redeploy the Agent container with the new command. + +{{% /tab %}} +{{< /tabs >}} + +(Optional) By default, SSI installs the latest SDK major versions. To pin specific versions, see [Set SDK tracer versions](#set-sdk-tracer-versions).
SSI adds a small amount of startup time to instrumented applications. If this overhead is not acceptable for your use case, contact Datadog Support.
-## Set SDK tracer versions +## Verify your first trace + +1. Restart your application containers so SSI can inject the Datadog SDK. Unlike a host installation, Docker injection applies to already-running containers when you restart them. +1. Generate traffic to your application. +1. In Datadog, go to [**APM** > **Traces**][5] to confirm that your service is reporting traces. -By default, Single Step Instrumentation installs the latest major versions of Datadog SDKs. Minor version updates are applied automatically when they become available. +If traces don't appear, check the following inside an instrumented container: -You may want to customize SDK versions based on your application's language version or specific environment requirements. You can control the major and minor versions used by customizing library versions during setup. +- Confirm that the injector is registered. `cat /etc/ld.so.preload` and the `LD_PRELOAD` environment variable should point to the Datadog launcher (`launcher.preload.so`). +- Set `DD_APM_INSTRUMENTATION_DEBUG=true`, then inspect `docker logs ` for injection messages. -To customize tracer versions: +For more help, see the [SSI troubleshooting guide][6]. + +## Set SDK tracer versions -1. In Datadog, go to the [Install the Datadog Agent on Docker][15] page. -1. After you turn on {{< ui >}}APM Instrumentation{{< /ui >}}, click {{< ui >}}Customize library versions{{< /ui >}}. +By default, Single Step Instrumentation installs the latest major versions of Datadog SDKs, and applies minor version updates automatically when they become available. - {{< img src="tracing/trace_collection/apm-instrumentation-version-pinning.png" alt="The 'Customize library versions' drop-down in the instructions for installing the Datadog Agent on Docker" style="width:100%;" >}} +To pin specific major or exact versions, set the `DD_APM_INSTRUMENTATION_LIBRARIES` variable in the install command: -1. Find your language(s) and use the dropdown to either: - - Pin an exact tracer version, or - - Select the major version you want to use. -1. Copy and run the updated installation command. +```shell +DD_API_KEY= DD_SITE= DD_APM_INSTRUMENTATION_ENABLED=docker \ + DD_APM_INSTRUMENTATION_LIBRARIES="java:1,python:3,js:5,dotnet:3,ruby:2,php:1" \ + bash -c "$(curl -L https://install.datadoghq.com/scripts/install_script_agent7.sh)" +``` -Available versions are listed in source repositories for each language: +Alternatively, use the in-app installer: after you turn on {{< ui >}}APM Instrumentation{{< /ui >}} on the [Install the Datadog Agent on Docker][4] page, click {{< ui >}}Customize library versions{{< /ui >}} and select an exact or major version for each language. -- [Java][8] (`java`) -- [Node.js][9] (`js`) -- [Python][10] (`python`) -- [.NET][11] (`dotnet`) -- [Ruby][12] (`ruby`) -- [PHP][13] (`php`) +Available versions are listed in source repositories for each language: [Java][8] (`java`), [Node.js][9] (`js`), [Python][10] (`python`), [.NET][11] (`dotnet`), [Ruby][12] (`ruby`), [PHP][13] (`php`). ## Configure Unified Service Tags -Unified Service Tags (USTs) apply consistent tags across traces, metrics, and logs, making it easier to navigate and correlate your observability data. Learn how to [set USTs for Docker services][16]. +Unified Service Tags (USTs) apply consistent tags across traces, metrics, and logs, making it easier to navigate and correlate your observability data. Learn how to [set USTs for Docker services][7]. ## Enable SDK-dependent products and features @@ -68,11 +87,11 @@ After SSI loads the Datadog SDK into your applications and enables distributed t {{< ssi-products >}} -To enable products, [set environment variables][3] in your application configuration. +To enable products, [set environment variables][14] in your application configuration. ## Remove Single Step APM instrumentation from your Agent -If you don't want to collect trace data for a particular service, host, VM, or container, complete the following steps: +If you don't want to collect trace data for a particular service, host, VM, or container, complete the following steps. ### Remove instrumentation for specific services @@ -100,25 +119,23 @@ To stop producing traces, uninstall APM and restart the infrastructure: ## Troubleshooting -If you encounter problems enabling APM with SSI, see the [SSI troubleshooting guide][17]. +If you encounter problems enabling APM with SSI, see the [SSI troubleshooting guide][6]. ## Further reading {{< partial name="whats-next/whats-next.html" >}} [1]: https://app.datadoghq.com/organization-settings/api-keys -[2]: /tracing/trace_collection/library_config/ -[3]: /tracing/trace_collection/library_config/ +[2]: /getting_started/site/ +[3]: /tracing/glossary/#instrumentation +[4]: https://app.datadoghq.com/fleet/install-agent/latest?platform=docker +[5]: https://app.datadoghq.com/apm/traces +[6]: /tracing/trace_collection/single-step-apm/troubleshooting/ +[7]: /getting_started/tagging/unified_service_tagging/?tab=docker#containerized-environment [8]: https://github.com/DataDog/dd-trace-java/releases [9]: https://github.com/DataDog/dd-trace-js/releases [10]: https://github.com/DataDog/dd-trace-py/releases [11]: https://github.com/DataDog/dd-trace-dotnet/releases [12]: https://github.com/DataDog/dd-trace-rb/releases [13]: https://github.com/DataDog/dd-trace-php/releases -[14]: /tracing/glossary/#instrumentation -[15]: https://app.datadoghq.com/fleet/install-agent/latest?platform=docker -[16]: /getting_started/tagging/unified_service_tagging/?tab=docker#containerized-environment -[17]: /tracing/trace_collection/automatic_instrumentation/single-step-apm/troubleshooting - - - +[14]: /tracing/trace_collection/library_config/ diff --git a/content/en/tracing/trace_collection/single-step-apm/kubernetes.md b/content/en/tracing/trace_collection/single-step-apm/kubernetes.md index 79cfa897ea7..680ca791881 100644 --- a/content/en/tracing/trace_collection/single-step-apm/kubernetes.md +++ b/content/en/tracing/trace_collection/single-step-apm/kubernetes.md @@ -30,11 +30,11 @@ Single Step Instrumentation (SSI) installs the Datadog Agent and instruments you - Kubernetes v1.20+ - [Helm][1] for deploying the Datadog Operator - [kubectl][2] for installing the Datadog Agent -- A supported language runtime per the [SSI compatibility guide][36] +- A supported language runtime per the [SSI compatibility guide][3] ### Check for existing tracer dependencies -SSI silently disables itself if it detects an existing tracer in your application. Before enabling SSI, check your dependency manifests and startup scripts: +SSI disables itself silently if it detects an existing tracer in your application. Before you enable SSI, check your dependency manifests and startup scripts: ```shell grep -rn "ddtrace\|dd-trace\|opentelemetry\|dd-java-agent\|javaagent" requirements.txt package.json Gemfile go.mod pom.xml build.gradle 2>/dev/null @@ -42,50 +42,14 @@ grep -rn "ddtrace\|dd-trace\|opentelemetry\|dd-java-agent\|javaagent" requiremen For Java, also check Dockerfiles and startup scripts for `-javaagent` flags, and check the `JAVA_TOOL_OPTIONS` environment variable. -If any matches are found, remove the tracer dependencies and rebuild your application image before proceeding. +If any matches are found, remove the tracer dependencies and rebuild your application image before you proceed. ### Compatibility notes -- **Node.js**: SSI requires CommonJS. If your application uses `import` syntax or sets `"type": "module"` in `package.json`, use [manually managed SDKs][44] instead. +- **Node.js**: SSI requires CommonJS. If your application uses `import` syntax or sets `"type": "module"` in `package.json`, use [manually managed SDKs][4] instead. - **Ruby**: SSI requires glibc and is not compatible with Alpine or other musl-based images. - **All other languages**: Alpine and musl-based images are supported on Kubernetes. SSI injects through the `LD_PRELOAD` environment variable, not `/etc/ld.so.preload`. -{{< agent-only >}} -Resolve these variables before starting: - -| Variable | How to resolve | -|---|---| -| CLUSTER_NAME | `kubectl config current-context` or `spec.global.clusterName` in datadog-agent.yaml | -| AGENT_NAMESPACE | Namespace where the Datadog Agent is installed (default: `datadog`) | -| APP_NAMESPACE | Namespace of the user's application — ask the user | -| DEPLOYMENT_NAME | `metadata.name` in the application Deployment — ask the user or check their repo | -| APP_LABEL | `spec.selector.matchLabels.app` in the Deployment | -| SERVICE_NAME | `tags.datadoghq.com/service` label on the Deployment — ask the user | -| ENV | `tags.datadoghq.com/env` label on the Deployment — ask the user | - -Pre-flight checks: -- `kubectl config current-context` returns the target cluster -- `helm version` confirms Helm is installed -- Check for existing Datadog deployment: `helm list -A | grep datadog` -- The user has a Datadog API key (do not ask them to share it in chat) -- Application language is one of: Java, Python, Ruby, Node.js, .NET, PHP -- Runtime version is within the SSI supported range per the compatibility guide -- For Node.js: confirm the app uses CommonJS, not ESM (`"type": "module"` in package.json means ESM) - -Check for existing instrumentation that silently disables SSI. Check both dependency manifests AND source files: -- Dependency manifests: `grep -rn "ddtrace\|dd-trace\|opentelemetry\|dd-java-agent\|javaagent" requirements.txt package.json Gemfile go.mod pom.xml build.gradle` -- Python: `grep -rn "import ddtrace\|from ddtrace\|ddtrace.patch_all" *.py` -- Node.js: `grep -rn "require.*dd-trace\|require.*ddtrace\|DD.init" *.js` -- Java: look for `-javaagent` in Dockerfile CMD/ENTRYPOINT or `JAVA_TOOL_OPTIONS` -- .NET: `grep -rn "Tracer.Instance\|DD.Trace" *.cs` -- Ruby: `grep -rn "require.*ddtrace\|Datadog.configure" *.rb` -- PHP: `grep -rn "DDTrace" *.php` - -Do NOT add `ddtrace-run` or `ddtrace` imports. SSI handles instrumentation automatically. Manual instrumentation causes SSI to silently disable itself. - -Always configure SSI through the DatadogAgent manifest (Helm values or Operator CR). Never use `kubectl patch`. Always use `kubectl apply -f` with a YAML file. -{{< /agent-only >}} - ## Enable SSI
SSI does not instrument applications in the namespace where the Datadog Agent is installed. Install the Agent in a separate namespace.
@@ -93,31 +57,9 @@ Always configure SSI through the DatadogAgent manifest (Helm values or Operator These steps enable SSI across your entire cluster. To instrument specific namespaces or pods, see [Target specific workloads](#target-specific-workloads). {{< tabs >}} -{{% tab "In-app wizard" %}} - -Datadog generates a configuration file with SSI enabled: - -1. Go to the [Install the Datadog Agent on Kubernetes][11] page. -1. Choose your installation method, select an API key, and set up the Operator or Helm repository. -1. Under **Configure `datadog-agent.yaml`**, go to **Additional configuration** > **Application Observability** and turn on **APM Instrumentation**. - - {{< img src="tracing/trace_collection/k8s-apm-instrumentation-toggle.jpg" alt="APM Instrumentation toggle in the Kubernetes Agent installation wizard" style="width:100%;" >}} - -1. Deploy the Agent with the generated configuration file. -1. Restart your application pods. - -
Restarting pods can cause a brief service interruption. Coordinate with the application owner before proceeding.
- - ```shell - kubectl rollout restart deployment/ -n - ``` - -[11]: https://app.datadoghq.com/fleet/install-agent/latest?platform=kubernetes - -{{% /tab %}} {{% tab "Helm" %}} -If the Datadog Agent is not installed, set `DD_API_KEY` to your [Datadog API key][45]. Then, add the Datadog Helm repository and create a Kubernetes Secret: +If the Datadog Agent is not installed, set `DD_API_KEY` to your [Datadog API key](https://app.datadoghq.com/organization-settings/api-keys). Then add the Datadog Helm repository and create a Kubernetes Secret: ```shell helm repo add datadog https://helm.datadoghq.com @@ -128,7 +70,7 @@ kubectl create secret generic datadog-secret --from-literal api-key=$DD_API_KEY If you install the Agent in a different namespace, replace `datadog` with your Agent namespace. -If the Agent is already installed, add `apm.instrumentation.enabled: true` to your existing `datadog-values.yaml` and skip to the `helm upgrade` step below. +If the Agent is already installed, add `apm.instrumentation.enabled: true` to your existing `datadog-values.yaml` and skip to the `helm upgrade` step. Create a `datadog-values.yaml`: @@ -142,7 +84,7 @@ datadog: enabled: true ``` -Replace `` with your Kubernetes cluster name and `` with your [Datadog site][46]. +Replace `` with your Kubernetes cluster name and `` with your [Datadog site](/getting_started/site/). Deploy or update the Agent: @@ -150,9 +92,11 @@ Deploy or update the Agent: helm upgrade --install datadog-agent -f datadog-values.yaml datadog/datadog -n datadog ``` +SSI installs the latest SDK for each supported language by default. To pin SDK versions, see [Pin SDK versions](#pin-sdk-versions). + Restart your application pods: -
Restarting pods can cause a brief service interruption. Coordinate with the application owner before proceeding.
+
Restarting pods can cause a brief service interruption. Coordinate with the application owner before you proceed.
```shell kubectl rollout restart deployment/ -n @@ -161,7 +105,7 @@ kubectl rollout restart deployment/ -n {{% /tab %}} {{% tab "Datadog Operator" %}} -If the Datadog Operator and Agent are not installed, set `DD_API_KEY` to your [Datadog API key][45]. Then, install the Operator and create a Kubernetes Secret: +If the Datadog Operator and Agent are not installed, set `DD_API_KEY` to your [Datadog API key](https://app.datadoghq.com/organization-settings/api-keys). Then install the Operator and create a Kubernetes Secret: ```shell helm repo add datadog https://helm.datadoghq.com @@ -172,7 +116,7 @@ kubectl create secret generic datadog-secret --from-literal api-key=$DD_API_KEY If you install the Agent in a different namespace, replace `datadog` with your Agent namespace. -If the Agent is already installed with the Datadog Operator, add `features.apm.instrumentation.enabled: true` to your existing `DatadogAgent` manifest and skip to the `kubectl apply` step below. +If the Agent is already installed with the Datadog Operator, add `features.apm.instrumentation.enabled: true` to your existing `DatadogAgent` manifest and skip to the `kubectl apply` step. Create a `datadog-agent.yaml`: @@ -196,7 +140,7 @@ spec: enabled: true ``` -Replace `` with your Kubernetes cluster name and `` with your [Datadog site][46]. +Replace `` with your Kubernetes cluster name and `` with your [Datadog site](/getting_started/site/). Apply the manifest: @@ -204,46 +148,48 @@ Apply the manifest: kubectl apply -f datadog-agent.yaml ``` +SSI installs the latest SDK for each supported language by default. To pin SDK versions, see [Pin SDK versions](#pin-sdk-versions). + Restart your application pods: -
Restarting pods can cause a brief service interruption. Coordinate with the application owner before proceeding.
+
Restarting pods can cause a brief service interruption. Coordinate with the application owner before you proceed.
```shell kubectl rollout restart deployment/ -n ``` {{% /tab %}} -{{< /tabs >}} +{{% tab "In-app wizard" %}} -
SSI adds a small amount of startup time to instrumented applications. If this overhead is not acceptable, contact Datadog Support.
+The Datadog in-app wizard generates a configuration file with SSI enabled: -
If your cluster enforces PodSecurity restricted policies, the Datadog init container may be blocked with: allowPrivilegeEscalation is false or violates PodSecurity "restricted:latest". See the SSI troubleshooting guide for the required security context configuration.
+1. Go to the [Install the Datadog Agent on Kubernetes](https://app.datadoghq.com/fleet/install-agent/latest?platform=kubernetes) page. +1. Choose your installation method, select an API key, and set up the Operator or Helm repository. +1. Under **Configure `datadog-agent.yaml`**, go to **Additional configuration** > **Application Observability** and turn on **APM Instrumentation**. -{{< agent-only >}} -Use one of the three tabs (In-app wizard, Helm, or Datadog Operator). For agents helping users, prefer the Helm or Operator tab. + {{< img src="tracing/trace_collection/k8s-apm-instrumentation-toggle.jpg" alt="APM Instrumentation toggle in the Kubernetes Agent installation wizard" style="width:100%;" >}} -Before changing configuration, check whether the Datadog Agent is already installed: -- `helm list -A | grep -i datadog` -- `kubectl get datadogagent -A` +1. Deploy the Agent with the generated configuration file. +1. Restart your application pods: -If the Agent is not installed, follow the bootstrap steps in the selected tab: add the Datadog Helm repository, create the `datadog` namespace, create `datadog-secret`, set `site` and `clusterName`, and deploy the Agent. +
Restarting pods can cause a brief service interruption. Coordinate with the application owner before you proceed.
-If the Agent is already installed, add `instrumentation.enabled: true` to the user's existing Helm values or `DatadogAgent` manifest. Do not create a separate manifest. + ```shell + kubectl rollout restart deployment/ -n + ``` -If the user wants to pin SDK versions, recommend these defaults: -`java: "1"`, `python: "3"`, `js: "5"`, `dotnet: "3"`, `ruby: "2"`, `php: "1"` +{{% /tab %}} +{{< /tabs >}} -After applying the config, confirm with the user before restarting their pods. After restart, wait for pods to be ready: -`kubectl wait --for=condition=Ready pod -l app= -n --timeout=120s` +
SSI adds a small amount of startup time to instrumented applications. If this overhead is not acceptable, contact Datadog Support.
-If `enabledNamespaces` or `disabledNamespaces` is set, SSI only instruments matching namespaces. You cannot set both. See the Target specific workloads section for workload targeting with `targets` and `podSelector`. -{{< /agent-only >}} +
If your cluster enforces PodSecurity restricted policies, the Datadog init container may be blocked with allowPrivilegeEscalation is false or violates PodSecurity "restricted:latest". See the SSI troubleshooting guide for the required security context configuration.
## Verify SSI is working -After restarting your application pods, wait two to three minutes for traces to arrive, then confirm SSI is working: +After you restart your application pods, wait two to three minutes for traces to arrive, then confirm SSI is working: -1. **Check for init containers.** Run this command against one of your application pods. The output should include `datadog-lib--init`: +1. **Check for the init container.** Run this command against one of your application pods. The output should include `datadog-lib--init`: ```shell kubectl get pod -n -o jsonpath='{.spec.initContainers[*].name}' ``` @@ -254,67 +200,18 @@ After restarting your application pods, wait two to three minutes for traces to curl http://localhost:8080/ ``` -3. **Find your service.** Go to [**APM** > **Services**][42] and confirm your service appears. - -4. **Confirm traces.** Go to [**APM** > **Traces**][43] and search for traces from your service. - -If traces do not appear after five minutes, see the [SSI troubleshooting guide][35]. - -{{< agent-only >}} -How SSI works (use this to reason about failures): -1. Admission webhook (registered by Cluster Agent) intercepts pod creation -2. Webhook mutates the pod spec — adds a `datadog-lib--init` init container -3. Init container copies the tracer library onto a shared volume -4. `LD_PRELOAD` env var is set pointing to the library .so file -5. Application process loads the library automatically on startup - -What each diagnostic layer can see: -- pup sees what Datadog's backend received. If pup shows no instrumented pods, the problem is in the cluster. -- kubectl sees cluster state. If kubectl shows the init container but pup shows no traces, the problem is post-injection. - -pup CLI setup (if not already available): -- Install: `brew tap datadog-labs/pack && brew install pup` -- Authenticate: `pup auth login` (opens browser for OAuth) -- Verify: `pup auth status` - -After the user restarts their pods, verify with these checks: - -1. Init containers: `kubectl get pod -n -o jsonpath='{.spec.initContainers[*].name}'` - Expected: includes `datadog-lib--init` - -2. If pup is available: - - `pup fleet instrumented-pods list ` - - `pup apm services list --env ` - - `pup traces search --query "service:" --from 1h --limit 5` - - `pup apm service-library-config get --service-name --env ` (validates tracer config) - -3. Runtime version (if compatibility is suspected): - - `kubectl exec -n -- python --version` - - `kubectl exec -n -- node --version` - - `kubectl exec -n -- java -version` - -No init container (injection never happened) — check: -- Namespace targeting: `kubectl get datadogagent datadog -n -o yaml | grep -A 15 instrumentation` -- Opt-out label: `kubectl get pod -n -o jsonpath='{.metadata.labels.admission\.datadoghq\.com/enabled}'` — if "false", the Admission Controller skips this pod -- Pod labels: `kubectl get pod -n --show-labels` -- Admission webhook registered: `kubectl get mutatingwebhookconfigurations | grep datadog` -- Cluster Agent running: `kubectl get pods -n -l app=datadog-cluster-agent` -- Whether the pod is in the Agent namespace (SSI skips its own namespace) - -Init container present but no traces (tracer not reporting) — check: -- Existing instrumentation in application code or dependency manifests -- Agent APM receiver: `kubectl exec -n $(kubectl get pod -n -l app=datadog-agent -o name | head -1) -- agent status | grep -A 5 "APM Agent"` -- Injection errors: `kubectl get pod -n -o jsonpath='{.spec.nodeName}'` then `pup apm troubleshooting list --hostname --timeframe 1h` -- Tracer status: `pup fleet tracers list --filter "service:"` -- Agent connectivity: `pup fleet agents list --filter "hostname:"` -{{< /agent-only >}} +3. **Find your service.** Go to [**APM** > **Services**][6] and confirm your service appears. + +4. **Confirm traces.** Go to [**APM** > **Traces**][7] and search for traces from your service. + +If traces do not appear after five minutes, see the [SSI troubleshooting guide][8]. ## Configure Unified Service Tags [Unified Service Tags][5] (USTs) apply consistent `service`, `env`, and `version` tags across traces, metrics, and logs. SSI can automatically extract UST values from your existing Kubernetes labels. -
-Automatic label extraction is not compatible with Remote Configuration. If you use Remote Configuration, configure USTs with ddTraceConfigs instead. +
+Automatic label extraction is not compatible with Remote Configuration. If you use Remote Configuration, configure USTs with ddTraceConfigs instead.
### Automatic label extraction (recommended) @@ -373,10 +270,21 @@ datadog: If your labels are not suitable for automatic extraction, set USTs directly in your deployment manifests with environment variables. This requires modifying each deployment individually. -For complete instructions, see [setting USTs for Kubernetes services][5]. +For complete instructions, see [setting USTs for Kubernetes services](/getting_started/tagging/unified_service_tagging/?tab=kubernetes#containerized-environment). {{% /collapse-content %}} +## Enable additional products + +After SSI enables distributed tracing, you can activate additional SDK-dependent products: + +{{< ssi-products >}} + +To enable products: + +- **With workload targeting (recommended):** Add `ddTraceConfigs` entries to your target blocks. See [Target specific workloads](#target-specific-workloads). +- **With environment variables:** Set variables directly in your application configuration. See [Library Configuration][9]. + ## Next steps After traces are flowing: @@ -388,297 +296,215 @@ After traces are flowing: {{< nextlink href="/tracing/trace_collection/dynamic_instrumentation/" >}}Dynamic Instrumentation: add custom spans without redeploying{{< /nextlink >}} {{< /whatsnext >}} -## Advanced SSI configuration +## Remove SSI -### Target specific workloads +### Remove instrumentation for specific services -By default, SSI instruments all services in all namespaces. Use one of the following methods to limit instrumentation scope. +Use [workload targeting](#target-specific-workloads) (Agent v7.64+) to exclude specific services. -{{< tabs >}} +Alternatively, add this label to the pod spec to skip Admission Controller mutation: -{{% tab "Agent v7.64+ (Recommended)" %}} +
This label disables all mutating webhooks for the pod, not only SSI.
-Define targeting blocks with `targets` to control which workloads are instrumented and what configuration they receive. +```yaml +spec: + template: + metadata: + labels: + admission.datadoghq.com/enabled: "false" +``` -| Key | Description | -|------------------|-------------| -| `name` | Target block name (metadata only). | -| `namespaceSelector` | Namespace(s) to instrument. Use `matchNames`, `matchLabels`, or `matchExpressions`. See the [Kubernetes selector documentation][10].| -| `podSelector` | Pod(s) to instrument. Use `matchLabels` or `matchExpressions`. See the [Kubernetes selector documentation][10]. | -| `ddTraceVersions` | [Datadog APM SDK][9] version per language. | -| `ddTraceConfigs` | SDK configuration: [Unified Service Tags][8], [additional products](#enable-additional-products), and [other APM settings][14]. | +Apply the change and restart the affected pods. -Edit `datadog-values.yaml` (Helm) or `datadog-agent.yaml` (Operator). Targets are evaluated in order; the first match wins. +### Remove instrumentation for all services -{{< collapse-content title="Example 1: Enable all namespaces except one" level="h4" >}} +{{< tabs >}} +{{% tab "Helm" %}} -{{< highlight yaml "hl_lines=4-10" >}} - apm: - instrumentation: - enabled: true - disabledNamespaces: - - "jenkins" - targets: - - name: "all-remaining-services" - ddTraceVersions: - java: "default" - python: "3.1.0" -{{< /highlight >}} +Set `instrumentation.enabled: false` in `datadog-values.yaml` and run: -{{< /collapse-content >}} +```shell +helm upgrade datadog-agent -f datadog-values.yaml datadog/datadog +``` -{{< collapse-content title="Example 2: Instrument specific namespaces by name and label" level="h4" >}} +{{% /tab %}} +{{% tab "Datadog Operator" %}} -{{< highlight yaml "hl_lines=4-28" >}} - apm: - instrumentation: - enabled: true - targets: - - name: "login-service_namespace" - namespaceSelector: - matchNames: - - "login-service" - ddTraceVersions: - java: "default" - ddTraceConfigs: - - name: "DD_PROFILING_ENABLED" - value: "auto" - - name: "billing-service_apps" - namespaceSelector: - matchLabels: - app: "billing-service" - ddTraceVersions: - python: "3.1.0" -{{< /highlight >}} - -{{< /collapse-content >}} - -{{< collapse-content title="Example 3: Different SDKs for different workloads" level="h4" >}} - -{{< highlight yaml "hl_lines=4-28" >}} - apm: - instrumentation: - enabled: true - targets: - - name: "db-user" - podSelector: - matchLabels: - app: "db-user" - ddTraceVersions: - java: "default" - ddTraceConfigs: - - name: "DD_DATA_STREAMS_ENABLED" - value: "true" - - name: "user-request-router" - podSelector: - matchLabels: - webserver: "user" - ddTraceVersions: - php: "default" -{{< /highlight >}} - -{{< /collapse-content >}} - -{{< collapse-content title="Example 4: Target a pod within a namespace" level="h4" >}} - -{{< highlight yaml "hl_lines=4-28" >}} - apm: - instrumentation: - enabled: true - targets: - - name: "login-service-namespace" - namespaceSelector: - matchNames: - - "login-service" - podSelector: - matchLabels: - app: "password-resolver" - ddTraceVersions: - java: "default" - ddTraceConfigs: - - name: "DD_PROFILING_ENABLED" - value: "auto" -{{< /highlight >}} - -{{< /collapse-content >}} - -{{< collapse-content title="Example 5: Exclude specific pods with matchExpressions" level="h4" >}} - -{{< highlight yaml "hl_lines=4-28" >}} - apm: - instrumentation: - enabled: true - targets: - - name: "default-target" - podSelector: - matchExpressions: - - key: app - operator: NotIn - values: - - app1 - - app2 -{{< /highlight >}} - -{{< /collapse-content >}} - -{{< collapse-content title="Example 6: Enable additional products" level="h4" >}} - -Enable [App and API Protection][12] and [Continuous Profiler][11] for a namespace: - -{{< highlight yaml "hl_lines=4-20" >}} - apm: - instrumentation: - enabled: true - targets: - - name: "web-apps-with-security" - namespaceSelector: - matchNames: - - "web-apps" - ddTraceVersions: - java: "default" - python: "default" - ddTraceConfigs: - - name: "DD_APPSEC_ENABLED" - value: "true" - - name: "DD_PROFILING_ENABLED" - value: "auto" -{{< /highlight >}} - -{{< /collapse-content >}} - -[8]: /getting_started/tagging/unified_service_tagging/?tab=kubernetes -[9]: /tracing/trace_collection/automatic_instrumentation/single-step-apm/compatibility/#tracer-libraries -[10]: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#resources-that-support-set-based-requirements -[11]: /profiler/ -[12]: /security/application_security/ -[14]: /tracing/trace_collection/library_config/ +Set `instrumentation.enabled: false` in `datadog-agent.yaml` and apply: -{{% /tab %}} +```shell +kubectl apply -f datadog-agent.yaml +``` -{{% tab "Agent <=v7.63 (Legacy)" %}} +{{% /tab %}} +{{< /tabs >}} -#### Namespace filtering +## Advanced configuration -Enable or disable instrumentation for specific namespaces. You can set `enabledNamespaces` or `disabledNamespaces`, but not both. +The following options tune SSI behavior after the happy path is working. Each is optional. -{{< collapse-content title="Datadog Operator" level="h5" >}} +{{% collapse-content title="Target specific workloads" level="h3" expanded=false id="target-specific-workloads" %}} -{{< highlight yaml "hl_lines=5-7" >}} - features: - apm: - instrumentation: - enabled: true - enabledNamespaces: - - default - - applications -{{< /highlight >}} +By default, SSI instruments all services in all namespaces. Use one of these methods to limit instrumentation scope. -{{< /collapse-content >}} +**Namespace filtering** -{{< collapse-content title="Helm" level="h5" >}} +Set `enabledNamespaces` or `disabledNamespaces` (mutually exclusive). Add these keys under `datadog.apm.instrumentation` (Helm) or `spec.features.apm.instrumentation` (Operator): -{{< highlight yaml "hl_lines=5-7" >}} - datadog: - apm: - instrumentation: - enabled: true - enabledNamespaces: - - namespace_1 - - namespace_2 -{{< /highlight >}} +```yaml +apm: + instrumentation: + enabled: true + enabledNamespaces: + - default + - applications +``` -{{< /collapse-content >}} +**Per-workload targeting (Agent v7.64+)** -#### SDK versions +Use `targets` to control which workloads are instrumented and what configuration they receive. Targets are evaluated in order; the first match wins. -Specify SDK versions to control which languages are instrumented and which library versions are used. You can set versions at the [service level](#specify-at-the-service-level) (pod annotations) or at the [cluster level](#specify-at-the-cluster-level) (Agent config). Service-level settings take precedence. +| Key | Description | +|---------------------|-------------| +| `name` | Target block name (metadata only). | +| `namespaceSelector` | Namespace(s) to instrument. Use `matchNames`, `matchLabels`, or `matchExpressions`. See the [Kubernetes selector documentation](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#resources-that-support-set-based-requirements). | +| `podSelector` | Pod(s) to instrument. Use `matchLabels` or `matchExpressions`. See the [Kubernetes selector documentation](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#resources-that-support-set-based-requirements). | +| `ddTraceVersions` | [Datadog APM SDK](/tracing/trace_collection/single-step-apm/compatibility/#supported-language-runtimes) version per language. | +| `ddTraceConfigs` | SDK configuration: Unified Service Tags, [additional products](#enable-additional-products), and [other APM settings](/tracing/trace_collection/library_config/). | -If you don't specify versions, all supported languages are instrumented with the latest SDK. +Example - instrument specific namespaces by name and label: -##### Specify at the service level +```yaml +apm: + instrumentation: + enabled: true + targets: + - name: login-service + namespaceSelector: + matchNames: + - login-service + ddTraceVersions: + java: "1" + ddTraceConfigs: + - name: DD_PROFILING_ENABLED + value: "auto" + - name: billing-service + namespaceSelector: + matchLabels: + app: billing-service + ddTraceVersions: + python: "3" +``` -Add a language annotation to your pod spec: +Example - exclude specific pods with `matchExpressions`: -| Language | Pod annotation | -|------------|-----------------------------------------------------------------------| -| Java | `admission.datadoghq.com/java-lib.version: ""` | -| Node.js | `admission.datadoghq.com/js-lib.version: ""` | -| Python | `admission.datadoghq.com/python-lib.version: ""` | -| .NET | `admission.datadoghq.com/dotnet-lib.version: ""` | -| Ruby | `admission.datadoghq.com/ruby-lib.version: ""` | -| PHP | `admission.datadoghq.com/php-lib.version: ""` | +```yaml +apm: + instrumentation: + enabled: true + targets: + - name: default-target + podSelector: + matchExpressions: + - key: app + operator: NotIn + values: + - app1 + - app2 +``` -Available versions: [Java][34], [Node.js][35], [Python][36], [.NET][37], [Ruby][38], [PHP][39]. +Example - enable additional products for a namespace: -
Exercise caution with the latest tag. Major releases may introduce breaking changes.
+```yaml +apm: + instrumentation: + enabled: true + targets: + - name: web-apps-with-security + namespaceSelector: + matchNames: + - web-apps + ddTraceVersions: + java: "1" + python: "3" + ddTraceConfigs: + - name: DD_APPSEC_ENABLED + value: "true" + - name: DD_PROFILING_ENABLED + value: "auto" +``` -##### Specify at the cluster level +**Legacy Agent (v7.63 and earlier)**: `targets` and `ddTraceVersions` are not available. Use `enabledNamespaces`/`disabledNamespaces` for namespace scope, and pin SDK versions with `libVersions` (cluster level) or the `admission.datadoghq.com/-lib.version` pod annotation. -{{< collapse-content title="Datadog Operator" level="h5" >}} +{{% /collapse-content %}} -{{< highlight yaml "hl_lines=5-8" >}} - features: - apm: - instrumentation: - enabled: true - libVersions: - dotnet: "x.x.x" - python: "x.x.x" - js: "x.x.x" -{{< /highlight >}} +{{% collapse-content title="Pin SDK versions" level="h3" expanded=false id="pin-sdk-versions" %}} -{{< /collapse-content >}} +SSI installs the latest SDK for each supported language by default. Use `ddTraceVersions` (Agent v7.64+, valid only inside a `targets` entry) to pin versions. Pinning reduces init container size, avoids downloading unnecessary SDKs, and makes tracer upgrades deliberate. -{{< collapse-content title="Helm" level="h5" >}} +```yaml +apm: + instrumentation: + enabled: true + targets: + - name: default-target + ddTraceVersions: + java: "1" + python: "3" + js: "5" + dotnet: "3" + ruby: "2" + php: "1" +``` -{{< highlight yaml "hl_lines=5-8" >}} - datadog: - apm: - instrumentation: - enabled: true - libVersions: - dotnet: "x.x.x" - python: "x.x.x" - js: "x.x.x" -{{< /highlight >}} +Specify a major version (for example, `python: "3"`) to receive the latest minor release, or an exact version to pin precisely. -{{< /collapse-content >}} +{{% /collapse-content %}} -[34]: https://github.com/DataDog/dd-trace-java/releases -[35]: https://github.com/DataDog/dd-trace-js/releases -[36]: https://github.com/DataDog/dd-trace-py/releases -[37]: https://github.com/DataDog/dd-trace-dotnet/releases -[38]: https://github.com/DataDog/dd-trace-rb/releases -[39]: https://github.com/DataDog/dd-trace-php/releases +{{% collapse-content title="Use opt-in labels for controlled rollout" level="h3" expanded=false %}} -{{% /tab %}} -{{< /tabs >}} +| Mode | Behavior | When to use | +| ------- | ----------- | ----------- | +| Default | All supported processes are instrumented. | Small clusters or prototypes. | +| Opt-in | [Workload targeting](#target-specific-workloads) restricts instrumentation to labeled pods. | Production clusters, staged rollouts, cost-sensitive environments. | -### Enable additional products +Add an opt-in label to your deployment and pod template: -After SSI enables distributed tracing, you can activate additional SDK-dependent products: +```yaml +metadata: + labels: + datadoghq.com/apm-instrumentation: "enabled" +``` -{{< ssi-products >}} +Then configure SSI to match: -To enable products: +```yaml +apm: + instrumentation: + enabled: true + targets: + - name: apm-instrumented + podSelector: + matchLabels: + datadoghq.com/apm-instrumentation: "enabled" +``` -- **With workload targeting (recommended):** Add `ddTraceConfigs` entries to your [target blocks](#target-specific-workloads). See [Example 6: Enable additional products](#target-specific-workloads). -- **With environment variables:** Set variables directly in your application configuration. See [Library Configuration][7]. +{{% /collapse-content %}} -### Configure injection modes +{{% collapse-content title="Configure injection modes" level="h3" expanded=false %}} SSI supports multiple injection modes that control how library files are delivered to application containers. Adjust this setting if you notice pod startup delays or high resource usage during initialization. | Mode | Description | Requirements | |------|-------------|--------------| | `init_container` | Copies files with an init container. | Agent deployed with Helm or Operator | -| `csi` | **In Preview.** Mounts files with the [Datadog CSI driver][37]. Faster pod startup. | Agent 7.76.0+, CSI driver 1.2.0+, Helm 3.178.1+ or Operator 1.25.0+ | +| `csi` | **In Preview.** Mounts files with the [Datadog CSI driver](/containers/kubernetes/csi_driver/). Faster pod startup. | Agent 7.76.0+, CSI driver 1.2.0+, Helm 3.178.1+ or Operator 1.25.0+ | -For `csi` mode, install and activate the CSI driver first. With Helm, set `datadog.csi.enabled: true`. See the [CSI driver documentation][37]. +For `csi` mode, install and activate the CSI driver first. With Helm, set `datadog.csi.enabled: true`. See the [CSI driver documentation](/containers/kubernetes/csi_driver/). -{{< collapse-content title="Set injection mode globally" level="h4" >}} +**Set injection mode globally** -**Helm** — add to `datadog-values.yaml`: +Helm - add to `datadog-values.yaml`: ```yaml datadog: apm: @@ -686,7 +512,7 @@ datadog: injectionMode: ``` -**Datadog Operator** — add to `datadog-agent.yaml`: +Datadog Operator - add to `datadog-agent.yaml`: ```yaml features: apm: @@ -696,12 +522,9 @@ features: Supported values: `init_container`, `csi`. -{{< /collapse-content >}} - -{{< collapse-content title="Set injection mode per pod" level="h4" >}} +**Set injection mode per pod** Add this annotation to the pod spec: - ```yaml metadata: annotations: @@ -710,9 +533,9 @@ metadata: Supported values: `init_container`, `csi`. -{{< /collapse-content >}} +{{% /collapse-content %}} -### Change the image registry +{{% collapse-content title="Change the image registry" level="h3" expanded=false %}} Datadog publishes SDK images on gcr.io (default), Docker Hub, and Amazon ECR. To use a different registry, set `DD_ADMISSION_CONTROLLER_AUTO_INSTRUMENTATION_CONTAINER_REGISTRY` in the Cluster Agent config: @@ -722,13 +545,13 @@ Datadog publishes SDK images on gcr.io (default), Docker Hub, and Amazon ECR. To | Docker Hub | `docker.io/datadog` | | Amazon ECR | `public.ecr.aws/datadog` | -For detailed instructions, see [Changing Your Container Registry][33]. +For detailed instructions, see [Changing Your Container Registry](/containers/guide/changing_container_registry/). -{{% collapse-content title="Use a private container registry" level="h4" expanded=false %}} +**Use a private container registry** If your organization cannot pull from public registries, mirror the Datadog images to your private registry: -1. [Mirror the images][34] for the languages you are instrumenting. At minimum, mirror `apm-inject` and the `dd-lib--init` images you need. +1. [Mirror the images](/containers/guide/sync_container_images/#copy-an-image-to-another-registry-using-crane) for the languages you are instrumenting. At minimum, mirror `apm-inject` and the `dd-lib--init` images you need. 2. Tag the images to match your configuration. If you set `ddTraceVersions` to `java: "1"` and `python: "3"`, mirror: - `apm-inject:0` @@ -739,7 +562,7 @@ If your organization cannot pull from public registries, mirror the Datadog imag {{% /collapse-content %}} -### Container Network Interface on EKS +{{% collapse-content title="Container Network Interface on EKS" level="h3" expanded=false %}} When using a CNI like Calico on EKS, control plane nodes cannot connect to the Admission Controller. Set `useHostNetwork: true` on the Cluster Agent: @@ -748,101 +571,11 @@ clusterAgent: useHostNetwork: true ``` -## Remove SSI - -### Remove instrumentation for specific services - -Use [workload targeting](#target-specific-workloads) (Agent v7.64+) to exclude specific services. - -Alternatively, add this label to the pod spec to skip Admission Controller mutation: - -
This label disables all mutating webhooks for the pod, not only SSI.
- -```yaml -spec: - template: - metadata: - labels: - admission.datadoghq.com/enabled: "false" -``` - -Apply the change and restart the affected pods. - -### Remove instrumentation for all services - -{{< tabs >}} -{{% tab "Helm" %}} - -Set `instrumentation.enabled: false` in `datadog-values.yaml` and run: - -```shell -helm upgrade datadog-agent -f datadog-values.yaml datadog/datadog -``` - -{{% /tab %}} -{{% tab "Datadog Operator" %}} - -Set `instrumentation.enabled: false` in `datadog-agent.yaml` and apply: - -```shell -kubectl apply -f datadog-agent.yaml -``` - -{{% /tab %}} -{{< /tabs >}} - -## SSI best practices - -{{% collapse-content title="Use opt-in labels for controlled rollout" level="h3" expanded=false %}} - -| Mode | Behavior | When to use | -| --- | ----------- | ----------- | -| Default | All supported processes are instrumented. | Small clusters or prototypes. | -| Opt-in | [Workload targeting](#target-specific-workloads) restricts instrumentation to labeled pods. | Production clusters, staged rollouts, cost-sensitive environments. | - -Add an opt-in label to your deployment and pod template: - -```yaml -metadata: - labels: - datadoghq.com/apm-instrumentation: "enabled" -``` - -Then configure SSI to match: - -```yaml -apm: - instrumentation: - enabled: true - targets: - - name: apm-instrumented - podSelector: - matchLabels: - datadoghq.com/apm-instrumentation: "enabled" -``` - -{{% /collapse-content %}} - -{{% collapse-content title="Pin SDK versions" level="h3" expanded=false %}} - -Use `ddTraceVersions` to control which SDK versions are injected. This reduces init container size, avoids downloading unnecessary SDKs, and makes tracer upgrades deliberate. - -```yaml -targets: - - name: default-target - ddTraceVersions: - java: "1" - python: "3" - js: "5" - php: "1" - dotnet: "3" -``` - {{% /collapse-content %}} ## Troubleshooting -If you encounter problems with SSI, see the [SSI troubleshooting guide][35]. +If you encounter problems with SSI, see the [SSI troubleshooting guide][8]. ## Further reading @@ -850,17 +583,10 @@ If you encounter problems with SSI, see the [SSI troubleshooting guide][35]. [1]: https://v3.helm.sh/docs/intro/install/ [2]: https://kubernetes.io/docs/tasks/tools/install-kubectl/ -[3]: /tracing/glossary/#instrumentation +[3]: /tracing/trace_collection/single-step-apm/compatibility/ +[4]: /tracing/trace_collection/dd_libraries/nodejs/ [5]: /getting_started/tagging/unified_service_tagging/?tab=kubernetes#containerized-environment -[7]: /tracing/trace_collection/library_config/ -[11]: https://app.datadoghq.com/fleet/install-agent/latest?platform=kubernetes -[33]: /containers/guide/changing_container_registry/ -[34]: /containers/guide/sync_container_images/#copy-an-image-to-another-registry-using-crane -[35]: /tracing/trace_collection/single-step-apm/troubleshooting -[36]: /tracing/trace_collection/single-step-apm/compatibility/ -[37]: /containers/kubernetes/csi_driver/ -[42]: https://app.datadoghq.com/apm/services -[43]: https://app.datadoghq.com/apm/traces -[44]: /tracing/trace_collection/dd_libraries/nodejs/ -[45]: https://app.datadoghq.com/organization-settings/api-keys -[46]: /getting_started/site/ +[6]: https://app.datadoghq.com/apm/services +[7]: https://app.datadoghq.com/apm/traces +[8]: /tracing/trace_collection/single-step-apm/troubleshooting/ +[9]: /tracing/trace_collection/library_config/ diff --git a/content/en/tracing/trace_collection/single-step-apm/linux.md b/content/en/tracing/trace_collection/single-step-apm/linux.md index 1c05450ef09..0ecc120ad41 100644 --- a/content/en/tracing/trace_collection/single-step-apm/linux.md +++ b/content/en/tracing/trace_collection/single-step-apm/linux.md @@ -13,95 +13,61 @@ further_reading: ## Overview -On a Linux host or VM, use Single Step Instrumentation (SSI) for APM to install the Datadog Agent and [instrument][14] your applications in one step, with no additional configuration required. +On a Linux host or VM, Single Step Instrumentation (SSI) installs the Datadog Agent and [instruments][1] your applications in one step, with no additional configuration required. -## Enable APM on your applications +Before you start, confirm that your environment is supported in the [SSI compatibility guide][2]. -
Before proceeding, confirm that your environment is compatible by reviewing the SSI compatibility guide.
+## Enable APM on your applications ### New Agent installation -If you don't yet have a Datadog Agent installed, follow these steps to install the Agent and enable SSI simultaneously. - -1. In Datadog, go to the [Install the Datadog Agent on Linux][15] page. -2. In the {{< ui >}}Customize your Agent coverage{{< /ui >}} section, go to {{< ui >}}Core Observability{{< /ui >}} and turn on {{< ui >}}Application Performance Monitoring{{< /ui >}}. - - {{< img src="tracing/trace_collection/enable_apm.png" alt="The 'Customize your Agent coverage' section of in-app instructions for installing the Datadog Agent on Linux" style="width:100%;" >}} - -{{< site-region region="us,us3,us5,eu,ap1,ap2,uk1" >}} -3. (Optional) By default, SSI installs the latest SDK versions. To use specific versions instead: +If you don't have a Datadog Agent installed, run the following command on your Linux host or VM to install the Agent and enable SSI at the same time: - Click {{< ui >}}Customize Library Versions{{< /ui >}}, then select your desired version for each language from the dropdowns. You can select an exact version or a major version, which uses the latest minor release available when the installation command is run. +```shell +DD_API_KEY= DD_SITE="{{< region-param key="dd_site" >}}" DD_APM_INSTRUMENTATION_ENABLED=host \ + bash -c "$(curl -L https://install.datadoghq.com/scripts/install_script_agent7.sh)" +``` - {{< img src="tracing/trace_collection/customize_library_versions.png" alt="The 'Customize library versions' drop-down in the instructions for installing the Datadog Agent on Linux" style="width:100%;" >}} - - Available versions are listed in source repositories for each language: [Java][8] (`java`), [Node.js][9] (`js`), [Python][10] (`python`), [.NET][11] (`dotnet`), [Ruby][12] (`ruby`), [PHP][13] (`php`). +Replace `` with your [Datadog API key][3]. After the command completes, restart your applications. -[8]: https://github.com/DataDog/dd-trace-java/releases -[9]: https://github.com/DataDog/dd-trace-js/releases -[10]: https://github.com/DataDog/dd-trace-py/releases -[11]: https://github.com/DataDog/dd-trace-dotnet/releases -[12]: https://github.com/DataDog/dd-trace-rb/releases -[13]: https://github.com/DataDog/dd-trace-php/releases +By default, SSI installs the latest SDK versions for all supported languages. -{{< /site-region >}} - -{{< site-region region="gov" >}} +#### Optional: pin SDK versions -3. (Optional) By default, SSI installs the latest SDK versions. To use specific versions instead, set your desired library versions with the `DD_APM_INSTRUMENTATION_LIBRARIES` variable in your Agent installation command: +To install specific SDK versions instead of the latest, add the `DD_APM_INSTRUMENTATION_LIBRARIES` variable: - ```shell - DD_API_KEY= - DD_SITE="US1-FED" - DD_APM_INSTRUMENTATION_ENABLED=host - DD_APM_INSTRUMENTATION_LIBRARIES="java:1,python:2,js:5,dotnet:3,php:1" - bash -c "$(curl -L https://install.datadoghq.com/scripts/install_script_agent7.sh)" - ``` +```shell +DD_API_KEY= DD_SITE="{{< region-param key="dd_site" >}}" DD_APM_INSTRUMENTATION_ENABLED=host \ + DD_APM_INSTRUMENTATION_LIBRARIES="java:1,python:3,js:5,dotnet:3,ruby:2,php:1" \ + bash -c "$(curl -L https://install.datadoghq.com/scripts/install_script_agent7.sh)" +``` - Available versions are listed in source repositories for each language: [Java][8] (`java`), [Node.js][9] (`js`), [Python][10] (`python`), [.NET][11] (`dotnet`), [Ruby][12] (`ruby`), [PHP][13] (`php`). +You can specify an exact version or a major version, which uses the latest minor release available when the command runs. Available versions are listed in the source repository for each language: [Java][4] (`java`), [Node.js][5] (`js`), [Python][6] (`python`), [.NET][7] (`dotnet`), [Ruby][8] (`ruby`), and [PHP][9] (`php`). -[8]: https://github.com/DataDog/dd-trace-java/releases -[9]: https://github.com/DataDog/dd-trace-js/releases -[10]: https://github.com/DataDog/dd-trace-py/releases -[11]: https://github.com/DataDog/dd-trace-dotnet/releases -[12]: https://github.com/DataDog/dd-trace-rb/releases -[13]: https://github.com/DataDog/dd-trace-php/releases +
SSI adds a small amount of startup time to instrumented applications. If this overhead is not acceptable for your use case, contact Datadog Support.
-{{< /site-region >}} +{{% collapse-content title="Alternate: install with the in-app wizard" level="h4" expanded=false %}} -{{< site-region region="gov2" >}} +To generate an installation command from the Datadog UI: -3. (Optional) By default, SSI installs the latest SDK versions. To use specific versions instead, set your desired library versions with the `DD_APM_INSTRUMENTATION_LIBRARIES` variable in your Agent installation command: +1. Go to the [Install the Datadog Agent on Linux][10] page. +1. In the {{< ui >}}Customize your Agent coverage{{< /ui >}} section, go to {{< ui >}}Core Observability{{< /ui >}} and turn on {{< ui >}}Application Performance Monitoring{{< /ui >}}. - ```shell - DD_API_KEY= - DD_SITE="US2-FED" - DD_APM_INSTRUMENTATION_ENABLED=host - DD_APM_INSTRUMENTATION_LIBRARIES="java:1,python:2,js:5,dotnet:3,php:1" - bash -c "$(curl -L https://install.datadoghq.com/scripts/install_script_agent7.sh)" - ``` + {{< img src="tracing/trace_collection/enable_apm.png" alt="The 'Customize your Agent coverage' section of in-app instructions for installing the Datadog Agent on Linux" style="width:100%;" >}} - Available versions are listed in source repositories for each language: [Java][8] (`java`), [Node.js][9] (`js`), [Python][10] (`python`), [.NET][11] (`dotnet`), [Ruby][12] (`ruby`), [PHP][13] (`php`). +1. (Optional) To pin SDK versions, click {{< ui >}}Customize Library Versions{{< /ui >}}, then select a version for each language from the dropdowns. -[8]: https://github.com/DataDog/dd-trace-java/releases -[9]: https://github.com/DataDog/dd-trace-js/releases -[10]: https://github.com/DataDog/dd-trace-py/releases -[11]: https://github.com/DataDog/dd-trace-dotnet/releases -[12]: https://github.com/DataDog/dd-trace-rb/releases -[13]: https://github.com/DataDog/dd-trace-php/releases + {{< img src="tracing/trace_collection/customize_library_versions.png" alt="The 'Customize library versions' drop-down in the instructions for installing the Datadog Agent on Linux" style="width:100%;" >}} -{{< /site-region >}} +1. Copy and run the generated command on your Linux host or VM, then restart your applications. -4. Copy and run the Agent installation command on your Linux host or VM. -5. Restart your applications. - -
SSI adds a small amount of startup time to instrumented applications. If this overhead is not acceptable for your use case, contact Datadog Support.
+{{% /collapse-content %}} ### Existing Agent installation -If you already have a Datadog Agent installed, use Fleet Automation to enable SSI. +If you already have a Datadog Agent installed, use Fleet Automation to enable SSI: -1. In Datadog, go to [**Fleet Automation > Configuration**][21]. +1. In Datadog, go to [**Fleet Automation > Configuration**][11]. 1. Click {{< ui >}}Configure Agents{{< /ui >}}. 1. Apply filters to select the agents you want to configure, then click **Next**. @@ -118,9 +84,40 @@ If you already have a Datadog Agent installed, use Fleet Automation to enable SS 1. Click **Next**. 1. Review your configuration and click {{< ui >}}Deploy Configuration{{< /ui >}}. +## Verify your first trace + +After you enable SSI, confirm that your applications are instrumented and sending traces: + +1. Restart your instrumented applications, then send requests to generate traffic. +1. Confirm that the Datadog libraries are loaded into a running process. Replace `` with the process ID of an instrumented application: + + ```shell + cat /proc//maps + ``` + + The output includes both the SSI launcher (`launcher.preload.so`) and a language library (for example, `libdd` or a language-specific tracer path). + +1. Confirm that the launcher is registered for preloading on the host: + + ```shell + cat /etc/ld.so.preload + ``` + + The output includes the path to `launcher.preload.so`. + +1. Check the APM section of the Agent status: + + ```shell + datadog-agent status + ``` + +1. In Datadog, go to [**APM > Service Catalog**](https://app.datadoghq.com/services) and confirm that your services appear. + +If your traces don't appear, see the [SSI troubleshooting guide][14]. + ## Configure Unified Service Tags -Unified Service Tags (USTs) apply consistent tags across traces, metrics, and logs, making it easier to navigate and correlate your observability data. Learn how to [set USTs for Linux services][16]. +Unified Service Tags (USTs) apply consistent tags across traces, metrics, and logs, making it easier to navigate and correlate your observability data. See [Set USTs for Linux services][12]. ## Enable SDK-dependent products and features @@ -130,15 +127,15 @@ After SSI loads the Datadog SDK into your applications and enables distributed t Use one of the following setup methods: -- **[Configure in `application_monitoring.yaml`][18]**: +- **[Configure in `application_monitoring.yaml`][13]**: Configure products and features across all services on a host without modifying application command lines. -- **[Set environment variables][17]**: +- **[Set environment variables][15]**: - Enable products by setting environment variables directly in your application configuration. + Enable products by setting environment variables directly in your application configuration. -## Advanced options +## Advanced configuration ### Update SDK version @@ -152,14 +149,14 @@ To update the SDK versions: ### Define instrumentation rules {{< site-region region="gov" >}} -
Instrumentation rules are not supported for your selected Datadog site ({{< region-param key="dd_site_name" >}}).
+
Instrumentation rules are not supported for your selected Datadog site ({{< region-param key="dd_site_name" >}}).
{{< /site-region >}} Instrumentation rules (available for Agent v7.73+) let you control which processes are automatically instrumented by SSI on Linux hosts. To configure instrumentation rules: -1. In Datadog, go to {{< ui >}}APM{{< /ui >}} > {{< ui >}}Service Setup{{< /ui >}} > [{{< ui >}}Manage Instrumentation Rules{{< /ui >}}][20]. +1. In Datadog, go to {{< ui >}}APM{{< /ui >}} > {{< ui >}}Service Setup{{< /ui >}} > [{{< ui >}}Manage Instrumentation Rules{{< /ui >}}][16]. 1. Click {{< ui >}}Add or Edit Rules{{< /ui >}}. 1. Define instrumentation rules: 1. Click {{< ui >}}Add New Rule{{< /ui >}}, then choose {{< ui >}}Allow Rule{{< /ui >}} or {{< ui >}}Block Rule{{< /ui >}} to specify whether matching processes should be instrumented. @@ -230,23 +227,25 @@ To stop producing traces for all services on your infrastructure: ## Troubleshooting -If you encounter problems enabling APM with SSI, see the [SSI troubleshooting guide][19]. +If you encounter problems enabling APM with SSI, see the [SSI troubleshooting guide][14]. ## Further reading {{< partial name="whats-next/whats-next.html" >}} -[8]: https://github.com/DataDog/dd-trace-java/releases -[9]: https://github.com/DataDog/dd-trace-js/releases -[10]: https://github.com/DataDog/dd-trace-py/releases -[11]: https://github.com/DataDog/dd-trace-dotnet/releases -[12]: https://github.com/DataDog/dd-trace-rb/releases -[13]: https://github.com/DataDog/dd-trace-php/releases -[14]: /tracing/glossary/#instrumentation -[15]: https://app.datadoghq.com/fleet/install-agent/latest?platform=linux -[16]: /getting_started/tagging/unified_service_tagging/?tab=kubernetes#non-containerized-environment -[17]: /tracing/trace_collection/library_config/ -[18]: /tracing/trace_collection/library_config/application_monitoring_yaml/ -[19]: /tracing/trace_collection/automatic_instrumentation/single-step-apm/troubleshooting -[20]: https://app.datadoghq.com/apm/service-setup/workload-selection -[21]: https://app.datadoghq.com/fleet/agent-management +[1]: /tracing/glossary/#instrumentation +[2]: /tracing/trace_collection/single-step-apm/compatibility/ +[3]: https://app.datadoghq.com/organization-settings/api-keys +[4]: https://github.com/DataDog/dd-trace-java/releases +[5]: https://github.com/DataDog/dd-trace-js/releases +[6]: https://github.com/DataDog/dd-trace-py/releases +[7]: https://github.com/DataDog/dd-trace-dotnet/releases +[8]: https://github.com/DataDog/dd-trace-rb/releases +[9]: https://github.com/DataDog/dd-trace-php/releases +[10]: https://app.datadoghq.com/fleet/install-agent/latest?platform=linux +[11]: https://app.datadoghq.com/fleet/agent-management +[12]: /getting_started/tagging/unified_service_tagging/?tab=kubernetes#non-containerized-environment +[13]: /tracing/trace_collection/library_config/application_monitoring_yaml/ +[14]: /tracing/trace_collection/single-step-apm/troubleshooting/ +[15]: /tracing/trace_collection/library_config/ +[16]: https://app.datadoghq.com/apm/service-setup/workload-selection diff --git a/content/en/tracing/trace_collection/single-step-apm/troubleshooting.md b/content/en/tracing/trace_collection/single-step-apm/troubleshooting.md index 7aae7bf71d9..2a06a524ce5 100644 --- a/content/en/tracing/trace_collection/single-step-apm/troubleshooting.md +++ b/content/en/tracing/trace_collection/single-step-apm/troubleshooting.md @@ -14,11 +14,38 @@ further_reading: ## Overview -Single Step Instrumentation (SSI) helps instrument applications by automatically loading application processes with the Datadog SDKs. SSI works for applications running on Linux hosts, in container environments such as Kubernetes and Docker, and for .NET applications served by Windows IIS—without requiring changes to application dependencies or images. If you encounter issues enabling APM with SSI, use this guide to troubleshoot and resolve common problems. For further assistance, contact [Datadog Support][1]. +[Single Step Instrumentation (SSI)][2] instruments applications by automatically loading application processes with the Datadog SDKs. SSI works for applications running on Linux hosts, in container environments such as Kubernetes and Docker, and for .NET applications served by Windows IIS, without requiring changes to application dependencies or images. If you enabled SSI but don't see traces, use this guide to find and fix the cause. For further assistance, contact [Datadog Support][1]. -## Triage: no traces after enabling SSI? +## How SSI injection works -If you enabled SSI but don't see traces, work through these checks in order: +Understanding the injection mechanism helps you reason about where a failure occurs. + +**Kubernetes:** + +1. An admission webhook (registered by the Cluster Agent) intercepts pod creation. +2. The webhook mutates the pod spec, adding a `datadog-lib--init` init container. +3. The init container copies the tracer library onto a shared volume. +4. The `LD_PRELOAD` environment variable is set, pointing to the library `.so` file. +5. The application process loads the library automatically on startup. + +**Linux hosts and Docker:** + +1. The installer adds the Datadog launcher to `/etc/ld.so.preload`. +2. Each newly launched process preloads `launcher.preload.so`. +3. The launcher detects the runtime and loads the matching tracer library into the process. + +Two vantage points help you locate a failure: + +- Cluster or host state (`kubectl`, `/proc//maps`, `/etc/ld.so.preload`) shows whether injection was applied. +- The Datadog UI (**APM** > **Services** and **Traces**) shows what the backend received. + +If injection was applied but no traces reach the backend, the problem is post-injection: the tracer isn't reporting, the Agent can't be reached, or existing instrumentation took precedence. + +SSI fails silently in several cases. It produces no error when it detects existing instrumentation, when the runtime version is unsupported, when a process isn't restarted after SSI is enabled, or when a pod is opted out or runs in the Agent's namespace. The checks below surface these cases. + +## First checks: no traces after enabling SSI + +Work through these checks in order for your platform. ### All platforms @@ -30,73 +57,104 @@ If you enabled SSI but don't see traces, work through these checks in order: ``` For Java, also check Dockerfiles and startup scripts for `-javaagent` flags, and check the `JAVA_TOOL_OPTIONS` environment variable. Remove any matches and rebuild your application before proceeding. -3. **Is the runtime version supported?** Check the [SSI compatibility guide][13]. +3. **Is the runtime version supported?** Check the [SSI compatibility guide][3]. -4. **For Node.js: is your application using ECMAScript Modules (ESM)?** SSI does not support ESM. If your application uses `import` syntax or sets `"type": "module"` in `package.json`, use [manually managed SDKs][14] instead. +4. **For Node.js: is your application using ECMAScript Modules (ESM)?** SSI does not support ESM. If your application uses `import` syntax or sets `"type": "module"` in `package.json`, use [manually managed SDKs][4] instead. ### Kubernetes -5. **Is your application in the same namespace as the Datadog Agent?** SSI does not instrument pods in the Agent namespace. +First, confirm whether injection happened. Run this against one application pod. The output should include `datadog-lib--init`: + +```shell +kubectl get pod -n -o jsonpath='{.spec.initContainers[*].name}' +``` -6. **Is a namespace or pod selector filtering your application out?** Check your SSI configuration for `enabledNamespaces`, `disabledNamespaces`, or `podSelector` targets that may not match your application's namespace or labels. Also check for the `admission.datadoghq.com/enabled: "false"` label on the pod, which tells the Admission Controller to skip it. +#### No init container (injection never happened) -### Linux hosts +Injection was never applied to the pod. Check: + +- **Namespace and target configuration:** + ```shell + kubectl get datadogagent datadog -n -o yaml | grep -A 15 instrumentation + ``` + Confirm your `enabledNamespaces`, `disabledNamespaces`, or `podSelector` targets match your application's namespace and labels. +- **Opt-out label**: if this returns `false`, the Admission Controller skips the pod: + ```shell + kubectl get pod -n -o jsonpath='{.metadata.labels.admission\.datadoghq\.com/enabled}' + ``` +- **Pod labels** (confirm the pod matches your targeting selectors): + ```shell + kubectl get pod -n --show-labels + ``` +- **Admission webhook registered:** + ```shell + kubectl get mutatingwebhookconfigurations | grep datadog + ``` +- **Cluster Agent running:** + ```shell + kubectl get pods -n -l app=datadog-cluster-agent + ``` +- **Agent namespace**: SSI does not instrument pods in the namespace where the Datadog Agent runs. -7. **Is `/etc/ld.so.preload` configured?** SSI on Linux uses `/etc/ld.so.preload` to load the injector. If this file is missing or does not contain the Datadog launcher path, SSI is not active. Check: +#### Init container present but no traces (tracer not reporting) + +The library was injected but no traces arrive. Check: + +- **Existing instrumentation** in application code or dependency manifests. Manual instrumentation (a bundled tracer, `-javaagent`, or `ddtrace` imports) causes SSI to disable itself silently. +- **Agent APM receiver:** + ```shell + kubectl exec -n $(kubectl get pod -n -l app=datadog-agent -o name | head -1) -- agent status | grep -A 5 "APM Agent" + ``` +- **Cluster node the pod is scheduled on** (for correlating Agent connectivity): + ```shell + kubectl get pod -n -o jsonpath='{.spec.nodeName}' + ``` +- **Wrong service or environment** on the traces you do see indicates a Unified Service Tagging problem. See [Unified Service Tagging][7] and [Injection appears successful but traces are missing](#injection-appears-successful-but-traces-are-missing). + +#### Runtime version (when compatibility is suspected) + +```shell +kubectl exec -n -- python --version +kubectl exec -n -- node --version +kubectl exec -n -- java -version +``` + +Compare against the [SSI compatibility guide][3]. + +### Linux hosts and Docker + +1. **Is the launcher loaded into the process?** The authoritative check is the process memory map. A successfully instrumented process shows **both** the launcher (`launcher.preload.so`) and a language library: + ```shell + grep -E "launcher.preload.so|/opt/datadog/apm/library/" /proc//maps + ``` + If both appear, injection succeeded. If the launcher appears but no language library does, the runtime wasn't detected or is unsupported. If neither appears, the preload isn't active for this process; continue with the checks below. + +2. **Was the process started after SSI was enabled?** The preload applies only to processes launched after `/etc/ld.so.preload` was written. Compare the process start time against the file's modification time: + ```shell + ps -o lstart= -p + stat -c %y /etc/ld.so.preload + ``` + +3. **Is `/etc/ld.so.preload` configured?** SSI on Linux uses `/etc/ld.so.preload` to load the launcher. Check: ```shell cat /etc/ld.so.preload ``` The output should contain `/opt/datadog-packages/datadog-apm-inject/stable/inject/launcher.preload.so`. -8. **Is the application using musl libc or a static binary?** The SSI injector on Linux requires glibc. Applications on Alpine or other musl-based distributions, and statically linked binaries (common with Go), cannot be instrumented with SSI. - -9. **Is SELinux or AppArmor blocking the injector?** Security policies can prevent `/etc/ld.so.preload` from being read. Check `getenforce` (SELinux) or `dmesg | grep apparmor` (AppArmor) for denials. - -If none of these apply, continue with the detailed troubleshooting methods below. - -{{< agent-only >}} -All platforms — start with pup if available: -- `pup traces search --query "service:" --from 1h --limit 5` -- `pup apm services list --env ` -- `pup apm troubleshooting list --hostname --timeframe 1h` - -Kubernetes diagnostics: - -Resolve the node hostname first: -- `kubectl get pod -n -o jsonpath='{.spec.nodeName}'` → use as NODE_HOSTNAME below - -Triage (run simultaneously): -- `kubectl get pod -n -o jsonpath='{.spec.initContainers[*].name}'` -- `kubectl describe pod -n | grep -A 10 "Events:"` -- `kubectl get datadogagent datadog -n -o yaml | grep -A 15 instrumentation` -- `pup fleet instrumented-pods list ` - -Kubernetes diagnosis by signal: -| Signal | Likely cause | Next step | -|--------|-------------|-----------| -| No init container on pod | Injection never happened | Check namespace targeting, pod-selector, opt-out label, webhook registration, pod not restarted | -| Init container present, no traces | Injection attempted but tracer not reporting | Check for existing ddtrace/OTel dependencies, runtime version, Agent connectivity | -| Pod events show CrashLoopBackOff | Init container failure | Check existing instrumentation conflicts, runtime version | -| Traces arriving but wrong service/env | UST misconfiguration | Check Unified Service Tag labels on the Deployment | - -Linux host diagnostics: -- Check injector is armed: `cat /etc/ld.so.preload` (should contain launcher path) -- Check tracer loaded in process: `grep datadog /proc//maps` -- Check process started after SSI: compare `ps -o lstart -p ` against `/etc/ld.so.preload` modification time -- Check libc type: `ldd --version` (musl = incompatible) -- Check binary type: `file /proc//exe` (statically linked = incompatible) -- Check Agent APM receiver: `datadog-agent status | grep -A 5 "APM Agent"` -- Check security: `getenforce` (SELinux), `dmesg | grep apparmor` (AppArmor) - -Key silent failure modes (SSI produces no error for these): -- Existing ddtrace or OpenTelemetry instrumentation: SSI silently disables itself -- Unsupported runtime version: silently skipped -- Kubernetes: `admission.datadoghq.com/enabled: "false"` label on pod: webhook skips the pod -- Application not restarted after SSI enabled: injection happens at startup only -- Kubernetes: pod in the Datadog Agent namespace: SSI skips its own namespace -- Linux: musl libc or static binary: injector requires glibc -- Linux: SELinux/AppArmor blocking `/etc/ld.so.preload` reads -{{< /agent-only >}} +4. **Is the application using musl libc or a static binary?** The SSI launcher on Linux requires glibc. Applications on Alpine or other musl-based distributions, and statically linked binaries (common with Go), cannot be instrumented with SSI. Check the libc and binary type: + ```shell + ldd --version # musl indicates an incompatible libc + file /proc//exe # "statically linked" indicates an incompatible binary + ``` + +5. **Is SELinux or AppArmor blocking the launcher?** Security policies can prevent `/etc/ld.so.preload` from being read. Check `getenforce` (SELinux) or `dmesg | grep apparmor` (AppArmor) for denials. + +6. **Is the Agent APM receiver up?** + ```shell + datadog-agent status | grep -A 5 "APM Agent" + ``` + +If none of these apply, continue with the diagnostic methods below. ## Troubleshooting methods @@ -104,8 +162,8 @@ You can investigate injection issues in the Datadog UI with Fleet Automation, or ### Troubleshoot injection in Datadog Fleet Automation - Fleet Automation provides two types of instrumentation insights for SSI: + - **Process-level insights** show instrumentation status and SDK installation details for individual hosts or containers. - **Kubernetes cluster insights** provide a higher-level view of instrumentation across your clusters, helping you understand how SSI configuration and injection are applied at scale. @@ -119,11 +177,11 @@ This functionality is available for: - **Environments**: Linux hosts, containers, Kubernetes - Datadog Agent v7.68.2+ -#### View process-level insights +#### View process-level insights Use process-level insights to verify whether SSI has been correctly applied to your application processes and to identify any injection failures. -1. Navigate to [Fleet Automation][9]. +1. Navigate to [Fleet Automation][5]. 1. Use facets to filter down to relevant hosts: - `single_step_instrumentation` shows which hosts have SSI enabled or disabled. - `single_step_instrumentation_status` shows which hosts have encountered issues with service instrumentation. @@ -137,7 +195,7 @@ Use process-level insights to verify whether SSI has been correctly applied to y Use cluster-level insights to understand how SSI is configured and functioning across your Kubernetes clusters. These insights extend troubleshooting beyond individual processes to show how instrumentation is applied to workloads at the cluster level. -1. Navigate to [{{< ui >}}Fleet Automation{{< /ui >}}][9] > {{< ui >}}View Agents{{< /ui >}}, and select {{< ui >}}Kubernetes Clusters{{< /ui >}} in the upper-right corner. +1. Navigate to [{{< ui >}}Fleet Automation{{< /ui >}}][5] > {{< ui >}}View Agents{{< /ui >}}, and select {{< ui >}}Kubernetes Clusters{{< /ui >}} in the upper-right corner. 1. Select a cluster to view its details, including: - Whether the cluster is managed by Helm or the Datadog Operator - The Cluster Agent and Node Agent versions @@ -147,8 +205,8 @@ Use cluster-level insights to understand how SSI is configured and functioning a - The pods identified as instrumentation targets based on cluster configuration or pod-level annotations - The status of each targeted pod, including whether instrumentation succeeded - The SDKs injected into each pod, including language and version - - Whether each instrumented workload is generating traces -1. Hover over any status icon to see contextual details about the state of instrumentation or trace collection. + - Whether each instrumented workload is generating traces +1. Hover over any status icon to see contextual details about the state of instrumentation or trace collection. {{< img src="tracing/trace_collection/k8s-ssi-tab.png" alt="The Single Step Instrumentation tab for a Kubernetes cluster, showing the SSI config yaml and a list of instrumented pods" style="width:100%;" >}} @@ -158,18 +216,20 @@ If the Datadog UI does not show any instrumentation issues, or if you're trouble To confirm injection at the container level, check that: -1. `/etc/ld.so.preload` includes the following entry: +1. `/etc/ld.so.preload` includes the following entry: ``` /opt/datadog-packages/datadog-apm-inject/stable/inject/launcher.preload.so - ``` + ``` 2. The `LD_PRELOAD` environment variable is set to the same value. 3. The directory `/opt/datadog-packages/datadog-apm-inject` exists, with `stable` and `$version` subdirectories. 4. Language-specific directories exist (for example, `/opt/datadog/apm/library/java/` for Java). +For a running process, `/proc//maps` is the authoritative confirmation: it shows both `launcher.preload.so` and the loaded language library. See [Linux hosts and Docker](#linux-hosts-and-docker). + To enable debug logs during manual verification: 1. Set the following in your pod spec: - + {{< code-block lang="yaml" disable_copy="true" collapsible="true" >}} env: - name: DD_TRACE_DEBUG # debug logging for the SDK @@ -177,15 +237,14 @@ To enable debug logs during manual verification: - name: DD_APM_INSTRUMENTATION_DEBUG # debug logging for the injector value: "true" {{< /code-block >}} - -2. Delete the pod to enable debug logs during injection. +2. Delete the pod to enable debug logs during injection. ## Injector debug logs Injector debug logs show what the injector decided for each process: whether injection succeeded, was denied, or was skipped, and why. Enable them when [Fleet Automation](#troubleshoot-injection-in-datadog-fleet-automation) doesn't explain a failure, when you're diagnosing at the host or container level, or when you're collecting information for [Datadog Support][1]. -Injector debug logs are separate from tracer debug logs. Use injector logs to diagnose whether and how a tracer was injected; after injection, use [tracer debug logs][15] to diagnose the tracer running inside the process. +Injector debug logs are separate from tracer debug logs. Use injector logs to diagnose whether and how a tracer was injected; after injection, use [tracer debug logs][6] to diagnose the tracer running inside the process. ### Enable debug mode @@ -324,9 +383,9 @@ A policy prevented injection for this process. The preceding `Evaluating '` while `/opt/datadog-packages/datadog-apm-library-/` is empty. When this happens, injection finds no library to load and no traces appear. + +Confirm the on-disk state: + +```shell +ls /opt/datadog-packages/datadog-apm-library-/ +``` + +If the directory is empty or missing, remove the stale registration and reinstall: + +```shell +datadog-installer remove datadog-apm-library- +``` + +Then re-run your platform's SSI installation command. + +#### Errors after manual uninstallation of Agent files + +If you manually delete Agent files, you may see errors like: -If you manually delete agent files, you may see errors like: ``` ERROR: ld.so: object /opt/datadog/apm/inject/launcher.preload.so from /etc/ld.so.preload cannot be preloaded (cannot open shared object file): ignored ``` To properly uninstall SSI, follow the platform-specific instructions: -* [Kubernetes][1] -* [Docker][2] -* [Linux][3] -* [Windows][4] +* [Kubernetes][10] +* [Docker][11] +* [Linux][12] +* [Windows][13] #### Injection not working with rootless Docker @@ -439,12 +517,13 @@ When using rootless Docker, set `docker_socket` in `/etc/datadog-agent/inject/do #### Injection fails with statically linked launchers If a custom launcher is statically linked (common with Go), the preload library might not be invoked. Injection can still succeed if: + - The launcher's command line includes the language name - The launcher runs an intermediary dynamically linked program However, direct process launches from statically linked binaries are not injected. -### Kubernetes environments +### Kubernetes The Datadog Admission Controller must be deployed and configured before application pods are created; it cannot modify existing pods. @@ -485,25 +564,27 @@ The Cluster Agent logs warnings and errors for injection failures, typically fro Use the metric `datadog.cluster_agent.admission_webhooks.library_injection_errors` for further debugging. -##### Language annotation cannot be applied +#### Language annotation cannot be applied During setup, SSI detects the application language of your service and applies a service label in the form `internal.dd.datadoghq.com/service-name.detected_langs`. If the label cannot be applied, injection fails. -Sometimes, labeling errors occur because a service name breaks Kubernetes string limits ([63 characters][6]). For example: +Sometimes, labeling errors occur because a service name breaks Kubernetes string limits ([63 characters][14]). For example: + ``` languagedetection/patcher.go:231 in handleDeploymentEvent) | failed to handle deployment event: annotations: Invalid value: "internal.dd.datadoghq.com/dummy-python-container-long-long-long-long-long-x.detected_langs": name part must be no more than 63 characters ``` -String limit violations are common if service tags are not explicitly set through [Unified Service Tagging][8], in which case default image names are used. +String limit violations are common if service tags are not explicitly set through [Unified Service Tagging][7], in which case default image names are used. -##### Injection appears successful but traces are missing +#### Injection appears successful but traces are missing If logs show no issues but traces are missing, there may be an application-side misconfiguration. Verify that: + - Required annotations and labels are present. -- [Unified Service Tagging][8] is set up correctly. +- [Unified Service Tagging][7] is set up correctly. - Allow/deny lists for instrumentation rules are properly defined. -## Language-specific troubleshooting +## Language-specific issues ### Java @@ -513,7 +594,7 @@ The `JAVA_TOOL_OPTIONS` environment variable has a JVM-enforced limit of 1024 ch To avoid this issue, exclude the affected process from injection. -#### `JAVA_TOOL_OPTIONS` changes program output +#### `JAVA_TOOL_OPTIONS` changes program output When `JAVA_TOOL_OPTIONS` is set, the JVM prints a message to stdout, such as `Picked up JAVA_TOOL_OPTIONS: -Xmx1024m`. If a process reads and depends on this output, it may be affected. @@ -524,6 +605,7 @@ As of version 0.12.2, injection is skipped for `java -version` to avoid interfer By default, Single Step sets the `DD_SERVICE` environment variable, which applies a single service name across all web applications running on the same server (such as Tomcat or WebLogic). As a result, all sites report under the same name. Use one of the following options to enable **split-by-tags** so that each site reports under its own name: + - JVM system property: `-Ddd.trace.split-by-tags=servlet.context` - Environment variable: `DD_TRACE_SPLIT_BY_TAGS=servlet.context` @@ -547,7 +629,7 @@ Versions <=2.7.5 contain a pre-packaged protobuf dependency that can conflict wi If SSI annotations and init containers are present on the pod but no .NET traces arrive, another profiler may have precedence. Check `CORECLR_PROFILER` on the main container. If the value is not `{846F5F1C-F9AE-4B07-969E-05C26BC060D8}` (the Datadog .NET tracer CLSID), another profiler is loaded instead of Datadog. -Remove the conflicting `CORECLR_*` environment variables (and any `LD_PRELOAD` entries that reference the other profiler) from the source that injected them: another vendor's operator, init container, pod template, or Helm values. Then roll the pods. The [.NET CLR Profiling API allows only one subscriber per process][10]. +Remove the conflicting `CORECLR_*` environment variables (and any `LD_PRELOAD` entries that reference the other profiler) from the source that injected them: another vendor's operator, init container, pod template, or Helm values. Then roll the pods. The [.NET CLR Profiling API allows only one subscriber per process][15]. ## Collect diagnostic information for support @@ -584,18 +666,18 @@ Collect the following details if troubleshooting injection in a Kubernetes envir {{< partial name="whats-next/whats-next.html" >}} -[1]: /tracing/trace_collection/single-step-apm/kubernetes?tab=agentv764recommended#remove-apm-for-all-services-on-the-infrastructure -[2]: /tracing/trace_collection/single-step-apm/docker#remove-apm-for-all-services-on-the-infrastructure -[3]: /tracing/trace_collection/single-step-apm/linux#remove-single-step-apm-instrumentation-from-your-agent -[4]: /tracing/trace_collection/single-step-apm/windows#remove-single-step-apm-instrumentation-from-your-agent -[5]: /containers/guide/sync_container_images/#copy-an-image-to-another-registry-using-crane -[6]: https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/#syntax-and-character-set -[7]: https://datatracker.ietf.org/doc/html/rfc1035 -[8]: /getting_started/tagging/unified_service_tagging/ -[9]: https://app.datadoghq.com/fleet -[10]: /tracing/trace_collection/dd_libraries/dotnet-core/#installation-and-getting-started -[11]: /tracing/guide/injectors/ -[12]: /tracing/trace_collection/single-step-apm/#instrument-sdks-across-applications -[13]: /tracing/trace_collection/single-step-apm/compatibility/ -[14]: /tracing/trace_collection/dd_libraries/nodejs/ -[15]: /tracing/troubleshooting/tracer_debug_logs/ +[1]: /help/ +[2]: /tracing/trace_collection/single-step-apm/ +[3]: /tracing/trace_collection/single-step-apm/compatibility/ +[4]: /tracing/trace_collection/dd_libraries/nodejs/ +[5]: https://app.datadoghq.com/fleet +[6]: /tracing/troubleshooting/tracer_debug_logs/ +[7]: /getting_started/tagging/unified_service_tagging/ +[8]: /tracing/guide/injectors/ +[9]: /tracing/trace_collection/single-step-apm/#instrument-sdks-across-applications +[10]: /tracing/trace_collection/single-step-apm/kubernetes?tab=agentv764recommended#remove-apm-for-all-services-on-the-infrastructure +[11]: /tracing/trace_collection/single-step-apm/docker#remove-apm-for-all-services-on-the-infrastructure +[12]: /tracing/trace_collection/single-step-apm/linux#remove-single-step-apm-instrumentation-from-your-agent +[13]: /tracing/trace_collection/single-step-apm/windows#remove-single-step-apm-instrumentation-from-your-agent +[14]: https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/#syntax-and-character-set +[15]: /tracing/trace_collection/dd_libraries/dotnet-core/#installation-and-getting-started diff --git a/content/en/tracing/trace_collection/single-step-apm/windows.md b/content/en/tracing/trace_collection/single-step-apm/windows.md index 5cb4fbb0485..e9760192d84 100644 --- a/content/en/tracing/trace_collection/single-step-apm/windows.md +++ b/content/en/tracing/trace_collection/single-step-apm/windows.md @@ -13,40 +13,47 @@ further_reading: ## Overview -With Single Step Instrumentation (SSI), you can enable APM for your Java and .NET applications on Windows VMs using a single Datadog Agent installation command. +With Single Step Instrumentation (SSI), you enable APM for your Windows applications by adding options to a single Datadog Agent installation command. -## Enable APM on Windows +- Instrumenting .NET applications on IIS is generally available (Agent v7.67.1+, .NET SDK v3.19.0+). +- Host-wide instrumentation of Java and .NET applications is in Preview. -
Before proceeding, confirm that your environment is compatible by reviewing the SSI compatibility guide.
+
Before you begin, confirm that your environment is supported in the SSI compatibility guide.
-### New Agent installation +## Enable APM on Windows -If you don't yet have a Datadog Agent installed, follow these steps to install the Agent and enable SSI simultaneously. +### New Agent installation -You can enable APM on Windows in the following ways: -* Instrument only .NET applications on IIS -* Instrument all Java and .NET applications across your entire Windows host +If you don't yet have a Datadog Agent installed, install the Agent and enable SSI in one step. {{< tabs >}} -{{% tab "IIS" %}} +{{% tab "IIS (.NET)" %}} -To instrument only .NET applications running on IIS: +To instrument .NET applications running on IIS, run the following command in an elevated PowerShell session. Replace `` with your [Datadog API key][1] and `` with your [Datadog site][2] ({{< region-param key="dd_site" code="true" >}}): -1. In Datadog, go to [Install the Datadog Agent on Windows][1]. -1. In the {{< ui >}}Customize your observability coverage{{< /ui >}} section, toggle {{< ui >}}Application Performance Monitoring (APM){{< /ui >}}. -1. (Optional) Set your SDK version: - - By default, Single Step Instrumentation installs the latest supported version of the Datadog .NET SDK. If you need to pin a specific version: +```powershell +$p = Start-Process -Wait -PassThru msiexec -ArgumentList '/qn /norestart /i "https://windows-agent.datadoghq.com/datadog-agent-7-latest.amd64.msi" /log C:\Windows\SystemTemp\install-datadog.log APIKEY="" SITE="" DD_APM_INSTRUMENTATION_ENABLED="iis"' +if ($p.ExitCode -ne 0) { + Write-Host "msiexec failed with exit code $($p.ExitCode) please check the logs at C:\Windows\SystemTemp\install-datadog.log" -ForegroundColor Red +} +``` + +By default, SSI installs the latest supported version of the Datadog .NET SDK. To pin a specific version, add the `DD_APM_INSTRUMENTATION_LIBRARIES` property to the argument list, for example `DD_APM_INSTRUMENTATION_LIBRARIES="dotnet:3"`. + +After installation, the Agent loads the Datadog .NET SDK into supported application processes to enable distributed tracing. - 1. Under {{< ui >}}Instrumentation Configuration{{< /ui >}}, select {{< ui >}}Customize Library Versions{{< /ui >}}. - 1. Under .NET, choose the version you want to use. - -1. Copy and run the provided MSI install command on your Windows host. -1. Restart the IIS applications you want instrumented. (You do not need to restart the entire IIS server.) +{{< collapse-content title="Alternate: install with the in-app wizard" level="h4" >}} + +1. In Datadog, go to [Install the Datadog Agent on Windows][3]. +1. In the {{< ui >}}Customize your observability coverage{{< /ui >}} section, toggle {{< ui >}}Application Performance Monitoring (APM){{< /ui >}}. +1. (Optional) Under {{< ui >}}Instrumentation Configuration{{< /ui >}}, select {{< ui >}}Customize Library Versions{{< /ui >}}, then choose the .NET version to pin. +1. Copy and run the generated MSI installation command on your Windows host. -After installation, the Agent automatically loads the Datadog .NET SDK into supported application processes to enable distributed tracing. +{{< /collapse-content >}} -[1]: https://app.datadoghq.com/fleet/install-agent/latest?platform=windows +[1]: https://app.datadoghq.com/organization-settings/api-keys +[2]: /getting_started/site/ +[3]: https://app.datadoghq.com/fleet/install-agent/latest?platform=windows {{% /tab %}} @@ -57,25 +64,31 @@ After installation, the Agent automatically loads the Datadog .NET SDK into supp Host-wide instrumentation for Windows is in Preview and limited to Preview participants. The installation and configuration options described in this tab appear in Datadog only after you are enrolled. Request access to join the Preview.
-To instrument Java and .NET applications across your entire Windows host: +After you are enrolled, instrument Java and .NET applications across your entire Windows host by running the following command in an elevated PowerShell session. Replace `` with your [Datadog API key][1] and `` with your [Datadog site][2] ({{< region-param key="dd_site" code="true" >}}): -1. In Datadog, go to [Install the Datadog Agent on Windows][1]. -1. In the {{< ui >}}Customize your observability coverage{{< /ui >}} section, toggle {{< ui >}}Application Performance Monitoring (APM){{< /ui >}}. -1. (Optional) Set your SDK version: - - By default, Single Step Instrumentation installs the latest supported version of the Datadog .NET and Java SDK. If you need to pin a specific version: +```powershell +$p = Start-Process -Wait -PassThru msiexec -ArgumentList '/qn /norestart /i "https://windows-agent.datadoghq.com/datadog-agent-7-latest.amd64.msi" /log C:\Windows\SystemTemp\install-datadog.log APIKEY="" SITE="" DD_APM_INSTRUMENTATION_ENABLED="host"' +if ($p.ExitCode -ne 0) { + Write-Host "msiexec failed with exit code $($p.ExitCode) please check the logs at C:\Windows\SystemTemp\install-datadog.log" -ForegroundColor Red +} +``` - 1. Under {{< ui >}}Instrumentation Configuration{{< /ui >}}, select {{< ui >}}Customize Library Versions{{< /ui >}}. - 1. Under .NET, choose the version you want to use. +By default, SSI installs the latest supported versions of the Datadog .NET and Java SDKs. To pin specific versions, add the `DD_APM_INSTRUMENTATION_LIBRARIES` property, for example `DD_APM_INSTRUMENTATION_LIBRARIES="dotnet:3,java:1"`. -1. Copy and run the provided MSI install command on your Windows host. -1. Configure instrumentation rules. +Host-wide SSI instruments all Java applications on the host and all .NET applications running in IIS. To instrument .NET applications running outside of IIS, or for granular control over which processes are instrumented, [define instrumentation rules](#define-instrumentation-rules). - Host-wide SSI automatically instruments all Java applications on the host and all .NET applications running in IIS. To instrument .NET applications running outside of IIS, you must [define an instrumentation rule](#define-instrumentation-rules) that allows them. You can also use instrumentation rules for granular control over which Java applications on the host or .NET applications in IIS are instrumented. +{{< collapse-content title="Alternate: install with the in-app wizard" level="h4" >}} + +1. In Datadog, go to [Install the Datadog Agent on Windows][3]. +1. In the {{< ui >}}Customize your observability coverage{{< /ui >}} section, toggle {{< ui >}}Application Performance Monitoring (APM){{< /ui >}}. +1. (Optional) Under {{< ui >}}Instrumentation Configuration{{< /ui >}}, select {{< ui >}}Customize Library Versions{{< /ui >}}, then choose the SDK versions to pin. +1. Copy and run the generated MSI installation command on your Windows host. -1. Restart the services you want instrumented. +{{< /collapse-content >}} -[1]: https://app.datadoghq.com/fleet/install-agent/latest?platform=windows +[1]: https://app.datadoghq.com/organization-settings/api-keys +[2]: /getting_started/site/ +[3]: https://app.datadoghq.com/fleet/install-agent/latest?platform=windows {{% /tab %}} {{< /tabs >}} @@ -102,13 +115,30 @@ If you already have a Datadog Agent installed, use Fleet Automation to enable SS 1. Click {{< ui >}}Next{{< /ui >}}. 1. Review your configuration and click {{< ui >}}Deploy Configuration{{< /ui >}}. -1. Configure instrumentation rules. - Host-wide SSI automatically instruments all Java applications on the host and all .NET applications running in IIS. To instrument .NET applications running outside of IIS, you must [define an instrumentation rule](#define-instrumentation-rules) that allows them. You can also use instrumentation rules for granular control over which Java applications on the host or .NET applications in IIS are instrumented. +For host-wide instrumentation, you can [define instrumentation rules](#define-instrumentation-rules) to instrument .NET applications running outside of IIS or to control which processes are instrumented. + +## Verify your first trace + +After you enable SSI, restart your application, generate traffic, and confirm that traces reach Datadog. + +For .NET on IIS: + +1. Recycle the IIS application pool for the app you want to instrument. A full server restart is not required: + + ```powershell + Restart-WebAppPool -Name "" + ``` + +1. Confirm that the Datadog profiler is loaded. In the instrumented worker process, `CORECLR_PROFILER` must equal `{846F5F1C-F9AE-4B07-969E-05C26BC060D8}`. If another profiler takes precedence, no Datadog traces flow. Remove any conflicting `CORECLR_*` environment variables so the Datadog profiler is used. +1. Generate traffic to your application. +1. In Datadog, confirm that your service appears in the [APM Services page][7]. + +If traces don't appear, see the [SSI troubleshooting guide][4]. ## Configure Unified Service Tags -Unified Service Tags (USTs) apply consistent tags across traces, metrics, and logs, making it easier to navigate and correlate your observability data. Learn how to [set USTs for Windows services][2]. +Unified Service Tags (USTs) apply consistent tags across traces, metrics, and logs, so you can navigate and correlate your observability data. Learn how to [set USTs for Windows services][2]. ## Enable SDK-dependent products and features @@ -118,16 +148,24 @@ After SSI loads the Datadog SDK into your applications and enables distributed t To enable products, [set environment variables][3] in your application configuration. -## Advanced options +## Remove Single Step APM instrumentation from your Agent + +To disable SSI for .NET on your host, run: + +```shell +&"C:\Program Files\Datadog\Datadog Agent\bin\datadog-installer.exe" remove datadog-apm-library-dotnet +``` -### Define instrumentation rules +## Advanced configuration + +{{< collapse-content title="Define instrumentation rules" level="h3" id="define-instrumentation-rules" >}} {{< site-region region="gov" >}}
Instrumentation rules are not supported for your selected Datadog site ({{< region-param key="dd_site_name" >}}).
{{< /site-region >}} {{< site-region region="us,us3,us5,eu,ap1,ap2,uk1" >}} -
Instrumentation rules (available for Agent v7.73+) apply only to host-wide instrumentation. They are not supported for IIS-only installation.
+
Instrumentation rules require Agent v7.73+ and apply only to host-wide instrumentation. They are not supported for IIS-only installation.
{{< /site-region >}} Instrumentation rules let you control which processes are automatically instrumented by SSI on Windows hosts. Rules are required to instrument .NET applications running outside of IIS. They are also useful for granular control over which Java applications on the host or .NET applications in IIS are instrumented. @@ -186,19 +224,13 @@ Instrument all processes by default. Add block rules to exclude services that wo {{< collapse-content title="Example 2: Instrument only specific IIS applications" level="h5" >}} -Block all instrumentation by default. Add allow rules to opt specific IIS applications into APM. Because all IIS workers share the w3wp.exe executable, use {{< ui >}}IIS Application Pool{{< /ui >}} to identify target applications. This approach is useful for gradual rollouts. +Block all instrumentation by default. Add allow rules to opt specific IIS applications into APM. Because all IIS workers share the `w3wp.exe` executable, use {{< ui >}}IIS Application Pool{{< /ui >}} to identify target applications. This approach is useful for gradual rollouts. {{< img src="tracing/trace_collection/instrumentation-rules-example-2.png" alt="Two allow instrumentation rules targeting specific IIS application pools by name, with a default of block instrumentation" style="width:100%;" >}} {{< /collapse-content >}} -## Remove Single Step APM instrumentation from your Agent - -To disable SSI for .NET on your host, run: - -```shell -&"C:\Program Files\Datadog\Datadog Agent\bin\datadog-installer.exe" remove datadog-apm-library-dotnet -``` +{{< /collapse-content >}} ## Troubleshooting @@ -208,9 +240,9 @@ If you encounter problems enabling APM with SSI, see the [SSI troubleshooting gu {{< partial name="whats-next/whats-next.html" >}} -[1]: https://app.datadoghq.com/fleet/install-agent/latest?platform=windows [2]: /integrations/windows-service/#tags [3]: /tracing/trace_collection/library_config/ -[4]: /tracing/trace_collection/automatic_instrumentation/single-step-apm/troubleshooting +[4]: /tracing/trace_collection/single-step-apm/troubleshooting/ [5]: https://app.datadoghq.com/apm/service-setup/workload-selection [6]: https://app.datadoghq.com/fleet/agent-management +[7]: https://app.datadoghq.com/apm/services From 09453b2dfdb54303886d230a8b4dcf22ccd6d9fa Mon Sep 17 00:00:00 2001 From: Brett Blue Date: Fri, 17 Jul 2026 15:49:17 -0600 Subject: [PATCH 11/13] Clarify single-step APM setup workflows --- content/en/tracing/trace_collection/_index.md | 14 +- .../single-step-apm/_index.md | 2 +- .../single-step-apm/compatibility.md | 41 +++- .../single-step-apm/docker.md | 58 +++-- .../single-step-apm/kubernetes.md | 224 +++++++++++++----- .../trace_collection/single-step-apm/linux.md | 16 +- .../single-step-apm/troubleshooting.md | 142 +++++++---- .../single-step-apm/windows.md | 23 +- 8 files changed, 353 insertions(+), 167 deletions(-) diff --git a/content/en/tracing/trace_collection/_index.md b/content/en/tracing/trace_collection/_index.md index 8bc0d43ba79..20fe1bbcea5 100644 --- a/content/en/tracing/trace_collection/_index.md +++ b/content/en/tracing/trace_collection/_index.md @@ -46,7 +46,7 @@ SSI automatically installs and loads Datadog SDKs into your application processe {{< nextlink href="/tracing/trace_collection/single-step-apm/kubernetes" >}}Kubernetes{{< /nextlink >}} {{< nextlink href="/tracing/trace_collection/single-step-apm/linux" >}}Linux{{< /nextlink >}} {{< nextlink href="/tracing/trace_collection/single-step-apm/docker" >}}Docker{{< /nextlink >}} - {{< nextlink href="/tracing/trace_collection/single-step-apm/windows" >}}Windows IIS{{< /nextlink >}} + {{< nextlink href="/tracing/trace_collection/single-step-apm/windows" >}}Windows{{< /nextlink >}} {{< /whatsnext >}} ## Need more control? @@ -69,18 +69,6 @@ After you have APM running, you can add custom spans for visibility into applica {{< nextlink href="/tracing/trace_collection/custom_instrumentation/" >}}Add custom spans with code-based instrumentation{{< /nextlink >}} {{< /whatsnext >}} -{{< agent-only >}} -If a user asks to set up APM, tracing, or distributed tracing, start with Single Step Instrumentation (SSI) unless they specifically request manual SDK control. - -First, ask what language they're using. If the language is Go, C++, or Rust, SSI is not available — direct them to manually managed SDKs at /tracing/trace_collection/dd_libraries/. - -Then ask what platform they're running on: -- Kubernetes → /tracing/trace_collection/single-step-apm/kubernetes -- Linux host or VM → /tracing/trace_collection/single-step-apm/linux -- Docker → /tracing/trace_collection/single-step-apm/docker -- Windows IIS → /tracing/trace_collection/single-step-apm/windows -{{< /agent-only >}} - ## Further reading {{< partial name="whats-next/whats-next.html" >}} diff --git a/content/en/tracing/trace_collection/single-step-apm/_index.md b/content/en/tracing/trace_collection/single-step-apm/_index.md index b61f9b8c107..f455ab0da4c 100644 --- a/content/en/tracing/trace_collection/single-step-apm/_index.md +++ b/content/en/tracing/trace_collection/single-step-apm/_index.md @@ -36,7 +36,7 @@ To learn more about how it works, see the [injector guide for Single Step Instru ## Prerequisites -1. Remove any custom instrumentation code from your application and restart it. SSI is automatically disabled if custom instrumentation is detected. +1. Check whether the application already loads a tracing SDK. Remove or disable only conflicting tracer initialization before enabling SSI, then restart the application. You can keep custom spans and non-tracing OpenTelemetry dependencies. 1. Confirm environment compatibility by reviewing the [SSI compatibility guide][5] for supported languages, operating systems, and architectures. ## Instrument SDKs across applications diff --git a/content/en/tracing/trace_collection/single-step-apm/compatibility.md b/content/en/tracing/trace_collection/single-step-apm/compatibility.md index ff5111a39d5..28e852482fe 100644 --- a/content/en/tracing/trace_collection/single-step-apm/compatibility.md +++ b/content/en/tracing/trace_collection/single-step-apm/compatibility.md @@ -1,5 +1,6 @@ --- title: Compatibility +description: "Review the supported operating systems, deployment environments, language runtimes, and limitations for Datadog Single Step Instrumentation." aliases: - /tracing/trace_collection/automatic_instrumentation/single-step-apm/compatibility/ further_reading: @@ -24,13 +25,13 @@ Select your environment to see compatibility requirements and limitations: ### Compatibility - **Status**: GA -- **Supported operating systems**: See [Linux distributions reference](#linux-distributions-reference) +- **Supported operating systems**: See [Linux distributions reference][linux-distributions] - **Supported architectures**: x86_64, arm64 ### Requirements - Datadog Agent with APM Instrumentation enabled -- A [supported Linux distribution](#linux-distributions-reference) +- A [supported Linux distribution][linux-distributions] ### Limitations @@ -44,13 +45,13 @@ Select your environment to see compatibility requirements and limitations: ### Compatibility - **Status**: GA -- **Supported operating systems**: See [Linux distributions reference](#linux-distributions-reference) +- **Supported operating systems**: See [Linux distributions reference][linux-distributions] - **Supported architectures**: x86_64, arm64 ### Requirements - Datadog Agent with APM Instrumentation enabled -- Docker running on a [supported Linux distribution](#linux-distributions-reference) +- Docker running on a [supported Linux distribution][linux-distributions] ### Limitations @@ -66,13 +67,13 @@ Select your environment to see compatibility requirements and limitations: ### Compatibility - **Status**: GA -- **Supported node pools**: Linux nodes only (see [Linux distributions reference](#linux-distributions-reference)) +- **Supported node pools**: Linux nodes only (see [Linux distributions reference][linux-distributions]) - **Supported architectures**: x86_64, arm64 ### Requirements - [Datadog Admission Controller][1] enabled -- Kubernetes nodes running a [supported Linux distribution](#linux-distributions-reference) +- Kubernetes nodes running a [supported Linux distribution][linux-distributions] ### Limitations @@ -100,6 +101,28 @@ Select your environment to see compatibility requirements and limitations: - **IIS only**: Only .NET applications running in IIS are supported. +{{% /tab %}} + +{{% tab "Windows Host-wide (Preview)" %}} + +### Compatibility + +- **Status**: Preview; enrollment is required +- **Supported runtimes**: Java and .NET + +### Requirements + +- Datadog Agent v7.73 or higher +- Access to the Windows host-wide instrumentation Preview +- For .NET applications outside IIS, an instrumentation rule that identifies the process to instrument + +### Limitations + +- Host-wide instrumentation is available only to Preview participants. +- Language support is limited to Java and .NET. + +For enrollment and setup instructions, see [Windows Single Step Instrumentation][windows-ssi]. + {{% /tab %}} {{< /tabs >}} @@ -234,10 +257,6 @@ Supported Node.js versions are available by default only on: For other distributions, you may need to install Node.js separately. -### Limitations - -- **ESM modules**: Instrumentation of ESM (ECMAScript modules) is not supported. - [1]: /tracing/trace_collection/compatibility/nodejs {{< /programming-lang >}} @@ -324,3 +343,5 @@ The following Linux distributions and architectures are supported for SSI across [1]: /containers/cluster_agent/admission_controller/ [2]: /tracing/guide/injectors/ +[windows-ssi]: /tracing/trace_collection/single-step-apm/windows/ +[linux-distributions]: #linux-distributions-reference diff --git a/content/en/tracing/trace_collection/single-step-apm/docker.md b/content/en/tracing/trace_collection/single-step-apm/docker.md index 866fecf7d66..486ce540c99 100644 --- a/content/en/tracing/trace_collection/single-step-apm/docker.md +++ b/content/en/tracing/trace_collection/single-step-apm/docker.md @@ -1,5 +1,6 @@ --- title: Single Step APM Instrumentation on Docker +description: "Enable Datadog Single Step Instrumentation for applications in Docker containers, verify traces, configure SDK versions and tags, and remove instrumentation." code_lang: docker type: multi-code-lang code_lang_weight: 10 @@ -13,23 +14,42 @@ further_reading: ## Overview -In a Docker Linux container, use Single Step Instrumentation (SSI) for APM to install the Datadog Agent and [instrument][3] your applications in one step, with no additional configuration required. +For applications in Docker Linux containers, Single Step Instrumentation (SSI) configures the Docker host to [instrument][3] supported application containers without changing their images. ## Enable APM on your applications -
Before you begin, confirm that your environment is compatible by reviewing the SSI compatibility guide.
+Before you begin: + +- Confirm that your environment meets the [SSI compatibility requirements][15]. +- Run the commands in this section on the Docker host, not inside the Agent or application container. {{< tabs >}} {{% tab "Installation command" %}} -Run the following command to install the Agent and enable SSI. Replace `` with your [Datadog API key][1] and `` with your [Datadog site][2]. +1. Configure the Docker host to inject Datadog SDKs into new containers: -```shell -DD_API_KEY= DD_SITE= DD_APM_INSTRUMENTATION_ENABLED=docker \ - bash -c "$(curl -L https://install.datadoghq.com/scripts/install_script_agent7.sh)" -``` + ```shell + DD_APM_INSTRUMENTATION_ENABLED=docker DD_NO_AGENT_INSTALL=true \ + bash -c "$(curl -L https://install.datadoghq.com/scripts/install_script_agent7.sh)" + ``` + + This installs the Datadog container injector and configures the Docker runtime. It does not install or replace the Agent container. + +1. Start the Agent with APM enabled and share the injector's socket directory. Replace `` with your [Datadog API key][1] and `` with your [Datadog site][2]: + + ```shell + docker run -d --name dd-agent \ + -e DD_API_KEY= \ + -e DD_SITE= \ + -e DD_APM_ENABLED=true \ + -e DD_APM_NON_LOCAL_TRAFFIC=true \ + -e DD_APM_RECEIVER_SOCKET=/opt/datadog/apm/inject/run/apm.socket \ + -v /opt/datadog/apm:/opt/datadog/apm \ + -v /var/run/docker.sock:/var/run/docker.sock:ro \ + registry.datadoghq.com/agent:7 + ``` -If the Agent is already running, redeploy the Agent container with the new command. + If the Agent container already exists, recreate it with equivalent APM socket and volume settings. Keep any other settings required by your environment. {{% /tab %}} {{% tab "In-app instructions" %}} @@ -39,25 +59,26 @@ If the Agent is already running, redeploy the Agent container with the new comma {{< img src="tracing/trace_collection/docker-apm-instrumentation-toggle.png" alt="The 'Customize your Agent install command' section of in-app instructions for installing the Datadog Agent on Docker" style="width:100%;" >}} -1. Copy and run the Agent installation command in your Docker container. If the Agent is already running, redeploy the Agent container with the new command. +1. Copy and run the generated commands on the Docker host. If the Agent is already running, recreate the Agent container with the generated configuration. {{% /tab %}} {{< /tabs >}} -(Optional) By default, SSI installs the latest SDK major versions. To pin specific versions, see [Set SDK tracer versions](#set-sdk-tracer-versions). +(Optional) By default, SSI installs the latest SDK major versions. To pin specific versions, see [Set SDK tracer versions][17].
SSI adds a small amount of startup time to instrumented applications. If this overhead is not acceptable for your use case, contact Datadog Support.
## Verify your first trace -1. Restart your application containers so SSI can inject the Datadog SDK. Unlike a host installation, Docker injection applies to already-running containers when you restart them. +1. Recreate or restart your application containers so SSI can inject the Datadog SDK when each process starts. 1. Generate traffic to your application. 1. In Datadog, go to [**APM** > **Traces**][5] to confirm that your service is reporting traces. -If traces don't appear, check the following inside an instrumented container: +If traces don't appear: -- Confirm that the injector is registered. `cat /etc/ld.so.preload` and the `LD_PRELOAD` environment variable should point to the Datadog launcher (`launcher.preload.so`). -- Set `DD_APM_INSTRUMENTATION_DEBUG=true`, then inspect `docker logs ` for injection messages. +- Confirm that the Agent and application containers share `/opt/datadog/apm` and that the Agent APM socket exists in that directory. +- Set `DD_APM_INSTRUMENTATION_DEBUG=true` on the application container, recreate it, then inspect `docker logs ` for injection messages. +- Inspect the application process environment inside the container for the language-specific settings described in the [injector guide][16]. For more help, see the [SSI troubleshooting guide][6]. @@ -68,8 +89,8 @@ By default, Single Step Instrumentation installs the latest major versions of Da To pin specific major or exact versions, set the `DD_APM_INSTRUMENTATION_LIBRARIES` variable in the install command: ```shell -DD_API_KEY= DD_SITE= DD_APM_INSTRUMENTATION_ENABLED=docker \ - DD_APM_INSTRUMENTATION_LIBRARIES="java:1,python:3,js:5,dotnet:3,ruby:2,php:1" \ +DD_APM_INSTRUMENTATION_ENABLED=docker DD_NO_AGENT_INSTALL=true \ + DD_APM_INSTRUMENTATION_LIBRARIES="java:1,python:4,js:5,dotnet:3,ruby:2,php:1" \ bash -c "$(curl -L https://install.datadoghq.com/scripts/install_script_agent7.sh)" ``` @@ -107,7 +128,7 @@ To remove APM instrumentation and stop sending traces from a specific service: To stop producing traces, uninstall APM and restart the infrastructure: -1. Run: +1. On the Docker host, run: ```shell dd-container-install --uninstall ``` @@ -139,3 +160,6 @@ If you encounter problems enabling APM with SSI, see the [SSI troubleshooting gu [12]: https://github.com/DataDog/dd-trace-rb/releases [13]: https://github.com/DataDog/dd-trace-php/releases [14]: /tracing/trace_collection/library_config/ +[15]: /tracing/trace_collection/single-step-apm/compatibility/ +[16]: /tracing/guide/injectors/#per-runtime-instrumentation +[17]: #set-sdk-tracer-versions diff --git a/content/en/tracing/trace_collection/single-step-apm/kubernetes.md b/content/en/tracing/trace_collection/single-step-apm/kubernetes.md index 680ca791881..518610c1582 100644 --- a/content/en/tracing/trace_collection/single-step-apm/kubernetes.md +++ b/content/en/tracing/trace_collection/single-step-apm/kubernetes.md @@ -28,25 +28,28 @@ Single Step Instrumentation (SSI) installs the Datadog Agent and instruments you ## Prerequisites - Kubernetes v1.20+ -- [Helm][1] for deploying the Datadog Operator -- [kubectl][2] for installing the Datadog Agent +- [Helm][1] for installing the Datadog Agent or Datadog Operator +- [kubectl][2] for applying manifests and restarting workloads - A supported language runtime per the [SSI compatibility guide][3] ### Check for existing tracer dependencies -SSI disables itself silently if it detects an existing tracer in your application. Before you enable SSI, check your dependency manifests and startup scripts: +Before you enable SSI, check whether your application already loads a tracing SDK. Search dependency manifests, lock files, Dockerfiles, and startup scripts: ```shell -grep -rn "ddtrace\|dd-trace\|opentelemetry\|dd-java-agent\|javaagent" requirements.txt package.json Gemfile go.mod pom.xml build.gradle 2>/dev/null +grep -RniE "ddtrace|dd-trace|dd-java-agent|javaagent|opentelemetry" \ + --include='requirements*.txt' --include='pyproject.toml' --include='poetry.lock' \ + --include='package*.json' --include='yarn.lock' --include='pnpm-lock.yaml' \ + --include='Gemfile*' --include='pom.xml' --include='build.gradle*' \ + --include='*.csproj' --include='Dockerfile*' . ``` For Java, also check Dockerfiles and startup scripts for `-javaagent` flags, and check the `JAVA_TOOL_OPTIONS` environment variable. -If any matches are found, remove the tracer dependencies and rebuild your application image before you proceed. +Review each match before making changes. Remove or disable only conflicting tracer initialization and dependencies, then rebuild the application image. Do not remove OpenTelemetry API, metrics, or logging dependencies solely because they match the search. ### Compatibility notes -- **Node.js**: SSI requires CommonJS. If your application uses `import` syntax or sets `"type": "module"` in `package.json`, use [manually managed SDKs][4] instead. - **Ruby**: SSI requires glibc and is not compatible with Alpine or other musl-based images. - **All other languages**: Alpine and musl-based images are supported on Kubernetes. SSI injects through the `LD_PRELOAD` environment variable, not `/etc/ld.so.preload`. @@ -54,12 +57,12 @@ If any matches are found, remove the tracer dependencies and rebuild your applic
SSI does not instrument applications in the namespace where the Datadog Agent is installed. Install the Agent in a separate namespace.
-These steps enable SSI across your entire cluster. To instrument specific namespaces or pods, see [Target specific workloads](#target-specific-workloads). +These steps enable SSI across your entire cluster. To instrument specific namespaces or pods, see [Target specific workloads][12]. {{< tabs >}} {{% tab "Helm" %}} -If the Datadog Agent is not installed, set `DD_API_KEY` to your [Datadog API key](https://app.datadoghq.com/organization-settings/api-keys). Then add the Datadog Helm repository and create a Kubernetes Secret: +If the Datadog Agent is not installed, set `DD_API_KEY` to your [Datadog API key][10]. Then add the Datadog Helm repository and create a Kubernetes Secret: ```shell helm repo add datadog https://helm.datadoghq.com @@ -70,7 +73,7 @@ kubectl create secret generic datadog-secret --from-literal api-key=$DD_API_KEY If you install the Agent in a different namespace, replace `datadog` with your Agent namespace. -If the Agent is already installed, add `apm.instrumentation.enabled: true` to your existing `datadog-values.yaml` and skip to the `helm upgrade` step. +If the Agent is already installed, add `datadog.apm.instrumentation.enabled: true` to your existing `datadog-values.yaml` and skip to the `helm upgrade` step. Create a `datadog-values.yaml`: @@ -84,7 +87,7 @@ datadog: enabled: true ``` -Replace `` with your Kubernetes cluster name and `` with your [Datadog site](/getting_started/site/). +Replace `` with your Kubernetes cluster name and `` with your [Datadog site][11]. Deploy or update the Agent: @@ -92,7 +95,7 @@ Deploy or update the Agent: helm upgrade --install datadog-agent -f datadog-values.yaml datadog/datadog -n datadog ``` -SSI installs the latest SDK for each supported language by default. To pin SDK versions, see [Pin SDK versions](#pin-sdk-versions). +SSI installs the latest SDK for each supported language by default. To pin SDK versions, see [Pin SDK versions][13]. Restart your application pods: @@ -102,10 +105,12 @@ Restart your application pods: kubectl rollout restart deployment/ -n ``` +For a StatefulSet or DaemonSet, replace `deployment/` with the controller type and name. + {{% /tab %}} {{% tab "Datadog Operator" %}} -If the Datadog Operator and Agent are not installed, set `DD_API_KEY` to your [Datadog API key](https://app.datadoghq.com/organization-settings/api-keys). Then install the Operator and create a Kubernetes Secret: +If the Datadog Operator and Agent are not installed, set `DD_API_KEY` to your [Datadog API key][10]. Then install the Operator and create a Kubernetes Secret: ```shell helm repo add datadog https://helm.datadoghq.com @@ -140,7 +145,7 @@ spec: enabled: true ``` -Replace `` with your Kubernetes cluster name and `` with your [Datadog site](/getting_started/site/). +Replace `` with your Kubernetes cluster name and `` with your [Datadog site][11]. Apply the manifest: @@ -148,7 +153,7 @@ Apply the manifest: kubectl apply -f datadog-agent.yaml ``` -SSI installs the latest SDK for each supported language by default. To pin SDK versions, see [Pin SDK versions](#pin-sdk-versions). +SSI installs the latest SDK for each supported language by default. To pin SDK versions, see [Pin SDK versions][13]. Restart your application pods: @@ -158,12 +163,14 @@ Restart your application pods: kubectl rollout restart deployment/ -n ``` +For a StatefulSet or DaemonSet, replace `deployment/` with the controller type and name. + {{% /tab %}} {{% tab "In-app wizard" %}} The Datadog in-app wizard generates a configuration file with SSI enabled: -1. Go to the [Install the Datadog Agent on Kubernetes](https://app.datadoghq.com/fleet/install-agent/latest?platform=kubernetes) page. +1. Go to the [Install the Datadog Agent on Kubernetes][14] page. 1. Choose your installation method, select an API key, and set up the Operator or Helm repository. 1. Under **Configure `datadog-agent.yaml`**, go to **Additional configuration** > **Application Observability** and turn on **APM Instrumentation**. @@ -183,16 +190,27 @@ The Datadog in-app wizard generates a configuration file with SSI enabled:
SSI adds a small amount of startup time to instrumented applications. If this overhead is not acceptable, contact Datadog Support.
-
If your cluster enforces PodSecurity restricted policies, the Datadog init container may be blocked with allowPrivilegeEscalation is false or violates PodSecurity "restricted:latest". See the SSI troubleshooting guide for the required security context configuration.
+
If your cluster uses init-container injection and enforces PodSecurity restricted policies, the Datadog init container may be blocked with allowPrivilegeEscalation is false or violates PodSecurity "restricted:latest". See the SSI troubleshooting guide for the required security context configuration.
## Verify SSI is working After you restart your application pods, wait two to three minutes for traces to arrive, then confirm SSI is working: -1. **Check for the init container.** Run this command against one of your application pods. The output should include `datadog-lib--init`: - ```shell - kubectl get pod -n -o jsonpath='{.spec.initContainers[*].name}' - ``` +1. **Check that the pod was mutated.** The expected evidence depends on the configured [injection mode][15]: + + - For `init_container`, the output of this command includes `datadog-lib--init`: + + ```shell + kubectl get pod -n -o jsonpath='{.spec.initContainers[*].name}' + ``` + + - For `csi` or `image_volume`, an SDK init container is not expected. Describe the pod and confirm that it has Datadog-injected volumes and mounts, and that its events contain no mount errors: + + ```shell + kubectl describe pod -n + ``` + + When `injectionMode` is `auto`, inspect the mutated pod to determine which delivery method the Cluster Agent selected. 2. **Generate traffic.** If your application does not receive traffic automatically, port-forward and send a few requests: ```shell @@ -200,7 +218,7 @@ After you restart your application pods, wait two to three minutes for traces to curl http://localhost:8080/ ``` -3. **Find your service.** Go to [**APM** > **Services**][6] and confirm your service appears. +3. **Find your service.** Go to [**APM** > **Catalog**][6] and confirm your service appears. 4. **Confirm traces.** Go to [**APM** > **Traces**][7] and search for traces from your service. @@ -216,39 +234,70 @@ Automatic label extraction is not compatible with }} + +Automatic label extraction requires Agent v7.69 or later and either Helm chart v3.120.0 or later or Datadog Operator v1.16.0 or later. {{% collapse-content title="Configure Unified Service Tags with ddTraceConfigs" level="h3" expanded=false %}} For granular control over specific workloads, use `ddTraceConfigs` to map labels to service configurations: +{{< tabs >}} +{{% tab "Helm" %}} + ```yaml datadog: - kubernetesResourcesLabelsAsTags: - pods: - app.kubernetes.io/name: service - deployments.apps: - app.kubernetes.io/name: service - tags: - - "env:production" apm: instrumentation: enabled: true @@ -262,15 +311,55 @@ datadog: valueFrom: fieldRef: fieldPath: metadata.labels['app.kubernetes.io/name'] + - name: DD_ENV + valueFrom: + fieldRef: + fieldPath: metadata.labels['app.kubernetes.io/environment'] + - name: DD_VERSION + valueFrom: + fieldRef: + fieldPath: metadata.labels['app.kubernetes.io/version'] ``` +{{% /tab %}} +{{% tab "Datadog Operator" %}} + +```yaml +spec: + features: + apm: + instrumentation: + enabled: true + targets: + - name: frontend-services + podSelector: + matchLabels: + tier: frontend + ddTraceConfigs: + - name: DD_SERVICE + valueFrom: + fieldRef: + fieldPath: metadata.labels['app.kubernetes.io/name'] + - name: DD_ENV + valueFrom: + fieldRef: + fieldPath: metadata.labels['app.kubernetes.io/environment'] + - name: DD_VERSION + valueFrom: + fieldRef: + fieldPath: metadata.labels['app.kubernetes.io/version'] +``` + +{{% /tab %}} +{{< /tabs >}} + {{% /collapse-content %}} {{% collapse-content title="Configure Unified Service Tags in deployment manifests" level="h3" expanded=false %}} If your labels are not suitable for automatic extraction, set USTs directly in your deployment manifests with environment variables. This requires modifying each deployment individually. -For complete instructions, see [setting USTs for Kubernetes services](/getting_started/tagging/unified_service_tagging/?tab=kubernetes#containerized-environment). +For complete instructions, see [setting USTs for Kubernetes services][5]. {{% /collapse-content %}} @@ -282,7 +371,7 @@ After SSI enables distributed tracing, you can activate additional SDK-dependent To enable products: -- **With workload targeting (recommended):** Add `ddTraceConfigs` entries to your target blocks. See [Target specific workloads](#target-specific-workloads). +- **With workload targeting (recommended):** Add `ddTraceConfigs` entries to your target blocks. See [Target specific workloads][12]. - **With environment variables:** Set variables directly in your application configuration. See [Library Configuration][9]. ## Next steps @@ -300,7 +389,7 @@ After traces are flowing: ### Remove instrumentation for specific services -Use [workload targeting](#target-specific-workloads) (Agent v7.64+) to exclude specific services. +Use [workload targeting][12] (Agent v7.64+) to exclude specific services. Alternatively, add this label to the pod spec to skip Admission Controller mutation: @@ -321,16 +410,16 @@ Apply the change and restart the affected pods. {{< tabs >}} {{% tab "Helm" %}} -Set `instrumentation.enabled: false` in `datadog-values.yaml` and run: +Set `datadog.apm.instrumentation.enabled: false` in `datadog-values.yaml` and run: ```shell -helm upgrade datadog-agent -f datadog-values.yaml datadog/datadog +helm upgrade -f datadog-values.yaml datadog/datadog -n ``` {{% /tab %}} {{% tab "Datadog Operator" %}} -Set `instrumentation.enabled: false` in `datadog-agent.yaml` and apply: +Set `spec.features.apm.instrumentation.enabled: false` in `datadog-agent.yaml` and apply: ```shell kubectl apply -f datadog-agent.yaml @@ -367,10 +456,10 @@ Use `targets` to control which workloads are instrumented and what configuration | Key | Description | |---------------------|-------------| | `name` | Target block name (metadata only). | -| `namespaceSelector` | Namespace(s) to instrument. Use `matchNames`, `matchLabels`, or `matchExpressions`. See the [Kubernetes selector documentation](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#resources-that-support-set-based-requirements). | -| `podSelector` | Pod(s) to instrument. Use `matchLabels` or `matchExpressions`. See the [Kubernetes selector documentation](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#resources-that-support-set-based-requirements). | -| `ddTraceVersions` | [Datadog APM SDK](/tracing/trace_collection/single-step-apm/compatibility/#supported-language-runtimes) version per language. | -| `ddTraceConfigs` | SDK configuration: Unified Service Tags, [additional products](#enable-additional-products), and [other APM settings](/tracing/trace_collection/library_config/). | +| `namespaceSelector` | Namespace(s) to instrument. Use `matchNames`, `matchLabels`, or `matchExpressions`. See the [Kubernetes selector documentation][16]. | +| `podSelector` | Pod(s) to instrument. Use `matchLabels` or `matchExpressions`. See the [Kubernetes selector documentation][16]. | +| `ddTraceVersions` | [Datadog APM SDK][17] version per language. | +| `ddTraceConfigs` | SDK configuration: Unified Service Tags, [additional products][18], and [other APM settings][9]. | Example - instrument specific namespaces by name and label: @@ -393,7 +482,7 @@ apm: matchLabels: app: billing-service ddTraceVersions: - python: "3" + python: "4" ``` Example - exclude specific pods with `matchExpressions`: @@ -426,7 +515,7 @@ apm: - web-apps ddTraceVersions: java: "1" - python: "3" + python: "4" ddTraceConfigs: - name: DD_APPSEC_ENABLED value: "true" @@ -450,14 +539,14 @@ apm: - name: default-target ddTraceVersions: java: "1" - python: "3" + python: "4" js: "5" dotnet: "3" ruby: "2" php: "1" ``` -Specify a major version (for example, `python: "3"`) to receive the latest minor release, or an exact version to pin precisely. +Specify a major version (for example, `python: "4"`) to receive the latest minor release, or an exact version to pin precisely. {{% /collapse-content %}} @@ -466,7 +555,7 @@ Specify a major version (for example, `python: "3"`) to receive the latest minor | Mode | Behavior | When to use | | ------- | ----------- | ----------- | | Default | All supported processes are instrumented. | Small clusters or prototypes. | -| Opt-in | [Workload targeting](#target-specific-workloads) restricts instrumentation to labeled pods. | Production clusters, staged rollouts, cost-sensitive environments. | +| Opt-in | [Workload targeting][12] restricts instrumentation to labeled pods. | Production clusters, staged rollouts, cost-sensitive environments. | Add an opt-in label to your deployment and pod template: @@ -497,10 +586,12 @@ SSI supports multiple injection modes that control how library files are deliver | Mode | Description | Requirements | |------|-------------|--------------| -| `init_container` | Copies files with an init container. | Agent deployed with Helm or Operator | -| `csi` | **In Preview.** Mounts files with the [Datadog CSI driver](/containers/kubernetes/csi_driver/). Faster pod startup. | Agent 7.76.0+, CSI driver 1.2.0+, Helm 3.178.1+ or Operator 1.25.0+ | +| `auto` | Lets the Cluster Agent select the delivery method supported by the cluster. | A Cluster Agent version that supports the selected method | +| `init_container` | Copies SDK files into the pod with an init container. | Agent deployed with Helm or Operator | +| `csi` | **Experimental.** Mounts SDK files with the [Datadog CSI driver][19]. | Cluster Agent 7.76.0+ and the Datadog CSI driver | +| `image_volume` | **Experimental.** Delivers SDK files with Kubernetes image volumes, without an SDK init container. | Cluster Agent 7.77.0+ and a cluster with image-volume support | -For `csi` mode, install and activate the CSI driver first. With Helm, set `datadog.csi.enabled: true`. See the [CSI driver documentation](/containers/kubernetes/csi_driver/). +For `csi` mode, install and activate the CSI driver first. With Helm, set `datadog.csi.enabled: true`. See the [CSI driver documentation][19]. **Set injection mode globally** @@ -520,7 +611,7 @@ features: injectionMode: ``` -Supported values: `init_container`, `csi`. +Supported values: `auto`, `init_container`, `csi`, `image_volume`. **Set injection mode per pod** @@ -531,32 +622,24 @@ metadata: admission.datadoghq.com/apm-inject.injection-mode: "" ``` -Supported values: `init_container`, `csi`. +Supported values: `auto`, `init_container`, `csi`, `image_volume`. {{% /collapse-content %}} {{% collapse-content title="Change the image registry" level="h3" expanded=false %}} -Datadog publishes SDK images on gcr.io (default), Docker Hub, and Amazon ECR. To use a different registry, set `DD_ADMISSION_CONTROLLER_AUTO_INSTRUMENTATION_CONTAINER_REGISTRY` in the Cluster Agent config: - -| Registry | Value | -|----------|-------| -| gcr.io (default) | `gcr.io/datadoghq` | -| Docker Hub | `docker.io/datadog` | -| Amazon ECR | `public.ecr.aws/datadog` | - -For detailed instructions, see [Changing Your Container Registry](/containers/guide/changing_container_registry/). +Datadog publishes SDK images to multiple public registries. The default can vary by Datadog site and cluster environment. For the current registry list and configuration options, see [Changing Your Container Registry][20]. **Use a private container registry** If your organization cannot pull from public registries, mirror the Datadog images to your private registry: -1. [Mirror the images](/containers/guide/sync_container_images/#copy-an-image-to-another-registry-using-crane) for the languages you are instrumenting. At minimum, mirror `apm-inject` and the `dd-lib--init` images you need. +1. [Mirror the images][21] for the languages you are instrumenting. At minimum, mirror `apm-inject` and the `dd-lib--init` images required by your configured injection mode. -2. Tag the images to match your configuration. If you set `ddTraceVersions` to `java: "1"` and `python: "3"`, mirror: +2. Tag the images to match your configuration. If you set `ddTraceVersions` to `java: "1"` and `python: "4"`, mirror: - `apm-inject:0` - `dd-lib-java-init:1` - - `dd-lib-python-init:3` + - `dd-lib-python-init:4` 3. Set `DD_ADMISSION_CONTROLLER_AUTO_INSTRUMENTATION_CONTAINER_REGISTRY` to your private registry URL. @@ -584,9 +667,20 @@ If you encounter problems with SSI, see the [SSI troubleshooting guide][8]. [1]: https://v3.helm.sh/docs/intro/install/ [2]: https://kubernetes.io/docs/tasks/tools/install-kubectl/ [3]: /tracing/trace_collection/single-step-apm/compatibility/ -[4]: /tracing/trace_collection/dd_libraries/nodejs/ [5]: /getting_started/tagging/unified_service_tagging/?tab=kubernetes#containerized-environment [6]: https://app.datadoghq.com/apm/services [7]: https://app.datadoghq.com/apm/traces [8]: /tracing/trace_collection/single-step-apm/troubleshooting/ [9]: /tracing/trace_collection/library_config/ +[10]: https://app.datadoghq.com/organization-settings/api-keys +[11]: /getting_started/site/ +[12]: #target-specific-workloads +[13]: #pin-sdk-versions +[14]: https://app.datadoghq.com/fleet/install-agent/latest?platform=kubernetes +[15]: #configure-injection-modes +[16]: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#resources-that-support-set-based-requirements +[17]: /tracing/trace_collection/single-step-apm/compatibility/#supported-language-runtimes +[18]: #enable-additional-products +[19]: /containers/kubernetes/csi_driver/ +[20]: /containers/guide/changing_container_registry/ +[21]: /containers/guide/sync_container_images/#copy-an-image-to-another-registry-using-crane diff --git a/content/en/tracing/trace_collection/single-step-apm/linux.md b/content/en/tracing/trace_collection/single-step-apm/linux.md index 0ecc120ad41..fa3090f3b7f 100644 --- a/content/en/tracing/trace_collection/single-step-apm/linux.md +++ b/content/en/tracing/trace_collection/single-step-apm/linux.md @@ -1,5 +1,6 @@ --- title: Single Step APM Instrumentation on Linux +description: "Enable Datadog Single Step Instrumentation on a Linux host or VM, verify traces, configure SDK versions and instrumentation rules, and remove instrumentation." code_lang: linux type: multi-code-lang code_lang_weight: 0 @@ -38,7 +39,7 @@ To install specific SDK versions instead of the latest, add the `DD_APM_INSTRUME ```shell DD_API_KEY= DD_SITE="{{< region-param key="dd_site" >}}" DD_APM_INSTRUMENTATION_ENABLED=host \ - DD_APM_INSTRUMENTATION_LIBRARIES="java:1,python:3,js:5,dotnet:3,ruby:2,php:1" \ + DD_APM_INSTRUMENTATION_LIBRARIES="java:1,python:4,js:5,dotnet:3,ruby:2,php:1" \ bash -c "$(curl -L https://install.datadoghq.com/scripts/install_script_agent7.sh)" ``` @@ -89,13 +90,13 @@ If you already have a Datadog Agent installed, use Fleet Automation to enable SS After you enable SSI, confirm that your applications are instrumented and sending traces: 1. Restart your instrumented applications, then send requests to generate traffic. -1. Confirm that the Datadog libraries are loaded into a running process. Replace `` with the process ID of an instrumented application: +1. Confirm that the SSI launcher is loaded into a running process. Replace `` with the process ID of an instrumented application: ```shell cat /proc//maps ``` - The output includes both the SSI launcher (`launcher.preload.so`) and a language library (for example, `libdd` or a language-specific tracer path). + The output includes `launcher.preload.so`. This confirms that the injector loaded; the tracer itself may be loaded through language-specific arguments or environment variables rather than as a mapped shared library. See [per-runtime instrumentation][17] for those mechanisms. 1. Confirm that the launcher is registered for preloading on the host: @@ -111,7 +112,7 @@ After you enable SSI, confirm that your applications are instrumented and sendin datadog-agent status ``` -1. In Datadog, go to [**APM > Service Catalog**](https://app.datadoghq.com/services) and confirm that your services appear. +1. In Datadog, go to [**APM > Catalog**][18] and confirm that your services appear. If your traces don't appear, see the [SSI troubleshooting guide][14]. @@ -161,7 +162,7 @@ To configure instrumentation rules: 1. Define instrumentation rules: 1. Click {{< ui >}}Add New Rule{{< /ui >}}, then choose {{< ui >}}Allow Rule{{< /ui >}} or {{< ui >}}Block Rule{{< /ui >}} to specify whether matching processes should be instrumented. 1. Name your rule. - 1. Add one or more conditions. See [Define rule conditions](#define-rule-conditions) to learn more. + 1. Add one or more conditions. See [Define rule conditions][19] to learn more. {{< img src="tracing/trace_collection/define_instrumentation_rule.png" alt="The instrumentation rules UI, showing configuration options for defining a rule" style="width:100%;" >}} @@ -244,8 +245,11 @@ If you encounter problems enabling APM with SSI, see the [SSI troubleshooting gu [9]: https://github.com/DataDog/dd-trace-php/releases [10]: https://app.datadoghq.com/fleet/install-agent/latest?platform=linux [11]: https://app.datadoghq.com/fleet/agent-management -[12]: /getting_started/tagging/unified_service_tagging/?tab=kubernetes#non-containerized-environment +[12]: /getting_started/tagging/unified_service_tagging/#non-containerized-environment [13]: /tracing/trace_collection/library_config/application_monitoring_yaml/ [14]: /tracing/trace_collection/single-step-apm/troubleshooting/ [15]: /tracing/trace_collection/library_config/ [16]: https://app.datadoghq.com/apm/service-setup/workload-selection +[17]: /tracing/guide/injectors/#per-runtime-instrumentation +[18]: https://app.datadoghq.com/services +[19]: #define-rule-conditions diff --git a/content/en/tracing/trace_collection/single-step-apm/troubleshooting.md b/content/en/tracing/trace_collection/single-step-apm/troubleshooting.md index 2a06a524ce5..54aab2416b3 100644 --- a/content/en/tracing/trace_collection/single-step-apm/troubleshooting.md +++ b/content/en/tracing/trace_collection/single-step-apm/troubleshooting.md @@ -23,21 +23,26 @@ Understanding the injection mechanism helps you reason about where a failure occ **Kubernetes:** 1. An admission webhook (registered by the Cluster Agent) intercepts pod creation. -2. The webhook mutates the pod spec, adding a `datadog-lib--init` init container. -3. The init container copies the tracer library onto a shared volume. -4. The `LD_PRELOAD` environment variable is set, pointing to the library `.so` file. -5. The application process loads the library automatically on startup. +2. The webhook mutates the pod spec to deliver the injector and SDKs. Depending on `injectionMode`, delivery uses init containers, the Datadog CSI driver, or Kubernetes image volumes. +3. The webhook configures the application container so the injector loads when the application process starts. +4. The injector detects the runtime and configures the matching tracer SDK. -**Linux hosts and Docker:** +**Linux hosts:** 1. The installer adds the Datadog launcher to `/etc/ld.so.preload`. 2. Each newly launched process preloads `launcher.preload.so`. -3. The launcher detects the runtime and loads the matching tracer library into the process. +3. The launcher detects the runtime and configures the matching tracer SDK. + +**Docker:** + +1. The installer configures Docker to use the Datadog `runc` shim. +2. When a new application container starts, the shim mounts the injector and SDK files and sets `LD_PRELOAD` in the container. +3. The launcher detects the runtime and configures the matching tracer SDK. Two vantage points help you locate a failure: -- Cluster or host state (`kubectl`, `/proc//maps`, `/etc/ld.so.preload`) shows whether injection was applied. -- The Datadog UI (**APM** > **Services** and **Traces**) shows what the backend received. +- Cluster, host, or container state shows whether injection was applied. +- The Datadog UI (**APM** > **Catalog** and **Traces**) shows what the backend received. If injection was applied but no traces reach the backend, the problem is post-injection: the tracer isn't reporting, the Agent can't be reached, or existing instrumentation took precedence. @@ -51,32 +56,53 @@ Work through these checks in order for your platform. 1. **Did you restart your application after enabling SSI?** SSI injects at startup. Existing processes and pods are not instrumented until restarted. -2. **Does your application have existing tracer dependencies?** SSI silently disables itself if it detects `ddtrace`, `dd-trace`, an OpenTelemetry SDK, or `-javaagent` in your application. Check your dependency manifests and startup scripts: +2. **Does your application already load a tracing SDK?** Check dependency manifests, lock files, Dockerfiles, and startup scripts: ```shell - grep -rn "ddtrace\|dd-trace\|opentelemetry\|dd-java-agent\|javaagent" requirements.txt package.json Gemfile go.mod pom.xml build.gradle 2>/dev/null + grep -RniE "ddtrace|dd-trace|dd-java-agent|javaagent|opentelemetry" \ + --include='requirements*.txt' --include='pyproject.toml' --include='poetry.lock' \ + --include='package*.json' --include='yarn.lock' --include='pnpm-lock.yaml' \ + --include='Gemfile*' --include='pom.xml' --include='build.gradle*' \ + --include='*.csproj' --include='Dockerfile*' . ``` - For Java, also check Dockerfiles and startup scripts for `-javaagent` flags, and check the `JAVA_TOOL_OPTIONS` environment variable. Remove any matches and rebuild your application before proceeding. + For Java, also check for `-javaagent` flags and inspect `JAVA_TOOL_OPTIONS`. Review each match: remove or disable only conflicting tracer initialization and dependencies, then rebuild the application. Do not remove OpenTelemetry API, metrics, or logging dependencies solely because they matched the search. 3. **Is the runtime version supported?** Check the [SSI compatibility guide][3]. -4. **For Node.js: is your application using ECMAScript Modules (ESM)?** SSI does not support ESM. If your application uses `import` syntax or sets `"type": "module"` in `package.json`, use [manually managed SDKs][4] instead. - ### Kubernetes -First, confirm whether injection happened. Run this against one application pod. The output should include `datadog-lib--init`: +First, determine which delivery method the pod uses. Inspect the pod annotation and your Helm or Operator configuration for `injectionMode`: + +```shell +kubectl get pod -n -o jsonpath='{.metadata.annotations.admission\.datadoghq\.com/apm-inject\.injection-mode}' +``` + +If the value is empty or `auto`, inspect the mutated pod to see which method the Cluster Agent selected: ```shell -kubectl get pod -n -o jsonpath='{.spec.initContainers[*].name}' +kubectl describe pod -n ``` -#### No init container (injection never happened) +- With `init_container`, the pod has a `datadog-lib--init` init container. +- With `csi` or `image_volume`, no SDK init container is expected; look for Datadog-injected volumes and mounts and check pod events for mount errors. + +#### Pod was not mutated Injection was never applied to the pod. Check: - **Namespace and target configuration:** + + For the Datadog Operator: + ```shell - kubectl get datadogagent datadog -n -o yaml | grep -A 15 instrumentation + kubectl get datadogagent -n -o yaml | grep -A 15 instrumentation ``` + + For Helm: + + ```shell + helm get values -n | grep -A 15 instrumentation + ``` + Confirm your `enabledNamespaces`, `disabledNamespaces`, or `podSelector` targets match your application's namespace and labels. - **Opt-out label**: if this returns `false`, the Admission Controller skips the pod: ```shell @@ -96,7 +122,7 @@ Injection was never applied to the pod. Check: ``` - **Agent namespace**: SSI does not instrument pods in the namespace where the Datadog Agent runs. -#### Init container present but no traces (tracer not reporting) +#### Pod was mutated but no traces arrive The library was injected but no traces arrive. Check: @@ -109,7 +135,7 @@ The library was injected but no traces arrive. Check: ```shell kubectl get pod -n -o jsonpath='{.spec.nodeName}' ``` -- **Wrong service or environment** on the traces you do see indicates a Unified Service Tagging problem. See [Unified Service Tagging][7] and [Injection appears successful but traces are missing](#injection-appears-successful-but-traces-are-missing). +- **Wrong service or environment** on the traces you do see indicates a Unified Service Tagging problem. See [Unified Service Tagging][7] and [Injection appears successful but traces are missing][16]. #### Runtime version (when compatibility is suspected) @@ -121,13 +147,13 @@ kubectl exec -n -- java -version Compare against the [SSI compatibility guide][3]. -### Linux hosts and Docker +### Linux hosts -1. **Is the launcher loaded into the process?** The authoritative check is the process memory map. A successfully instrumented process shows **both** the launcher (`launcher.preload.so`) and a language library: +1. **Is the launcher loaded into the process?** Check the process memory map: ```shell - grep -E "launcher.preload.so|/opt/datadog/apm/library/" /proc//maps + grep "launcher.preload.so" /proc//maps ``` - If both appear, injection succeeded. If the launcher appears but no language library does, the runtime wasn't detected or is unsupported. If neither appears, the preload isn't active for this process; continue with the checks below. + A match confirms that the launcher loaded. It does not, by itself, prove that the language tracer loaded; tracers can be configured through runtime arguments or environment variables. Use [injector debug logs][17] or inspect the [language-specific injection setting][18] to confirm that step. 2. **Was the process started after SSI was enabled?** The preload applies only to processes launched after `/etc/ld.so.preload` was written. Compare the process start time against the file's modification time: ```shell @@ -156,9 +182,31 @@ Compare against the [SSI compatibility guide][3]. If none of these apply, continue with the diagnostic methods below. +### Docker + +Docker injection uses a custom runtime shim and container-level `LD_PRELOAD`; it does not depend on the host's `/etc/ld.so.preload`. + +1. **Was the application container created after Docker SSI was enabled?** Recreate or restart the container so the Datadog runtime shim can inject it. +2. **Is the Agent container configured for SSI?** Confirm that it has APM enabled, uses `/opt/datadog/apm/inject/run/apm.socket`, and shares `/opt/datadog/apm` with the host. +3. **Was the injector added to the application process?** Inspect the process environment for `LD_PRELOAD` and the setting used by its language runtime: + + ```shell + docker exec sh -c 'tr "\0" "\n" < /proc/1/environ | grep -E "LD_PRELOAD|JAVA_TOOL_OPTIONS|NODE_OPTIONS|PYTHONPATH|CORECLR_"' + ``` + +4. **Is the Agent APM receiver up?** Run `agent status` in the Agent container and check the APM section: + + ```shell + docker exec agent status + ``` + +5. **What decision did the injector make?** Enable `DD_APM_INSTRUMENTATION_DEBUG=true` on the application container, recreate it, and inspect `docker logs `. + +For rootless Docker or custom runtime shims, also review the Docker limitations in the [SSI compatibility guide][3]. + ## Troubleshooting methods -You can investigate injection issues in the Datadog UI with Fleet Automation, or manually at the container level. For a lower-level view of what the injector decided for each process, see [Injector debug logs](#injector-debug-logs). +You can investigate injection issues in the Datadog UI with Fleet Automation, or manually at the container level. For a lower-level view of what the injector decided for each process, see [Injector debug logs][17]. ### Troubleshoot injection in Datadog Fleet Automation @@ -210,21 +258,15 @@ Use cluster-level insights to understand how SSI is configured and functioning a {{< img src="tracing/trace_collection/k8s-ssi-tab.png" alt="The Single Step Instrumentation tab for a Kubernetes cluster, showing the SSI config yaml and a list of instrumented pods" style="width:100%;" >}} -### Manually verify injection in the application container +### Manually verify injection -If the Datadog UI does not show any instrumentation issues, or if you're troubleshooting a single service or container, you can manually verify whether injection occurred as expected. This method is helpful when debugging in environments where centralized visibility is limited or when a specific service isn't reporting traces. +If the Datadog UI does not show any instrumentation issues, manually verify the stages that apply to your environment: -To confirm injection at the container level, check that: - -1. `/etc/ld.so.preload` includes the following entry: - ``` - /opt/datadog-packages/datadog-apm-inject/stable/inject/launcher.preload.so - ``` -2. The `LD_PRELOAD` environment variable is set to the same value. -3. The directory `/opt/datadog-packages/datadog-apm-inject` exists, with `stable` and `$version` subdirectories. -4. Language-specific directories exist (for example, `/opt/datadog/apm/library/java/` for Java). +- **Linux host:** Confirm that `/etc/ld.so.preload` contains `/opt/datadog-packages/datadog-apm-inject/stable/inject/launcher.preload.so` and that the injector package directory exists. +- **Docker:** Confirm that the application process has an `LD_PRELOAD` setting for the Datadog launcher and that the Agent and application containers can access the injected files and APM socket under `/opt/datadog/apm`. +- **Kubernetes:** Inspect the mutated pod. Init-container mode adds `datadog-lib--init`; CSI and image-volume modes add Datadog volumes and mounts without that SDK init container. -For a running process, `/proc//maps` is the authoritative confirmation: it shows both `launcher.preload.so` and the loaded language library. See [Linux hosts and Docker](#linux-hosts-and-docker). +On Linux-based platforms, finding `launcher.preload.so` in `/proc//maps` confirms that the injector loaded into that process. Confirm tracer configuration separately by checking the runtime-specific argument or environment variable in the [injector guide][18], or by reviewing injector debug logs. To enable debug logs during manual verification: @@ -242,7 +284,7 @@ To enable debug logs during manual verification: ## Injector debug logs -Injector debug logs show what the injector decided for each process: whether injection succeeded, was denied, or was skipped, and why. Enable them when [Fleet Automation](#troubleshoot-injection-in-datadog-fleet-automation) doesn't explain a failure, when you're diagnosing at the host or container level, or when you're collecting information for [Datadog Support][1]. +Injector debug logs show what the injector decided for each process: whether injection succeeded, was denied, or was skipped, and why. Enable them when [Fleet Automation][19] doesn't explain a failure, when you're diagnosing at the host or container level, or when you're collecting information for [Datadog Support][1]. Injector debug logs are separate from tracer debug logs. Use injector logs to diagnose whether and how a tracer was injected; after injection, use [tracer debug logs][6] to diagnose the tracer running inside the process. @@ -421,9 +463,7 @@ Datadog maintains an internal deny list to prevent injection into certain proces #### Linux instrumentation rules -{{< callout url="https://docs.google.com/forms/d/e/1FAIpQLSdMu6WAsUCD3djkl_oN0Qh7fQmBCiKYyUvuqlYWRyObebAc6Q/viewform" header="Join the Preview!">}} -Instrumentation rules are available for Linux-based apps through a limited availability preview. To configure allow or deny rules for process injection, sign up for preview access. -{{< /callout >}} +Instrumentation rules require Agent v7.73 or later. They let you allow or block Linux processes based on properties such as executable, arguments, working directory, and detected language. See [Define instrumentation rules on Linux][20]. #### Kubernetes instrumentation rules @@ -442,7 +482,7 @@ Datadog adheres to security best practices and works with security vendors to al ### Environments with strict pod security settings -If pod security rules block the Datadog init container, you may see errors like: +If pod security rules block the Datadog init container when using `init_container` mode, you may see errors like: ``` Privilege escalation container is not allowed or violates PodSecurity "restricted: latest": allowPrivilegeEscalation is false @@ -469,9 +509,9 @@ Setting `DD_TRACE_ENABLED=false` does not prevent SSI from loading the SDK. The #### Host injection does not apply to existing processes -The preload library only injects into newly launched processes. Start a new shell session or log out and log back in to apply instrumentation. +On a Linux host, the preload library only injects into newly launched processes. Restart the application process; if its environment comes from a login session, start a new shell session or log out and back in first. -**Note**: Docker-based injection does not have this limitation. +Docker does not depend on the host login session, but application containers still must be recreated or restarted after SSI is enabled. #### Injection fails on small instance types @@ -627,7 +667,7 @@ Versions <=2.7.5 contain a pre-packaged protobuf dependency that can conflict wi #### SSI is applied but no .NET traces reach the Agent -If SSI annotations and init containers are present on the pod but no .NET traces arrive, another profiler may have precedence. Check `CORECLR_PROFILER` on the main container. If the value is not `{846F5F1C-F9AE-4B07-969E-05C26BC060D8}` (the Datadog .NET tracer CLSID), another profiler is loaded instead of Datadog. +If the pod shows SSI mutation but no .NET traces arrive, another profiler may have precedence. Check `CORECLR_PROFILER` on the main container. If the value is not `{846F5F1C-F9AE-4B07-969E-05C26BC060D8}` (the Datadog .NET tracer CLSID), another profiler is loaded instead of Datadog. Remove the conflicting `CORECLR_*` environment variables (and any `LD_PRELOAD` entries that reference the other profiler) from the source that injected them: another vendor's operator, init container, pod template, or Helm values. Then roll the pods. The [.NET CLR Profiling API allows only one subscriber per process][15]. @@ -645,7 +685,7 @@ When contacting support about injection issues, collect the following informatio It should be owned by `root` with `644` permissions (`-rw-r--r--`). -1. Enable injector debug logs and collect the output. For instructions, see [Injector debug logs](#injector-debug-logs). +1. Enable injector debug logs and collect the output. For instructions, see [Injector debug logs][17]. 1. Provide an Agent flare. @@ -653,14 +693,14 @@ When contacting support about injection issues, collect the following informatio Collect the following details if troubleshooting injection in a Kubernetes environment: -- The method used to deploy the Cluster Agent (for example, Helm, Datadog Operator, or kubectl commands). +- The method used to deploy the Cluster Agent (for example, Helm or Datadog Operator) and the configured injection mode. - Deployment files for the application pod. - Flares from both the Node Agent and the Cluster Agent, ideally with `DEBUG` mode enabled. - Output of: ``` kubectl describe pod ``` -- Injector debug logs from the application pod (not the Cluster Agent). For instructions, see [Injector debug logs](#injector-debug-logs). +- Injector debug logs from the application pod (not the Cluster Agent). For instructions, see [Injector debug logs][17]. ## Further reading @@ -669,15 +709,19 @@ Collect the following details if troubleshooting injection in a Kubernetes envir [1]: /help/ [2]: /tracing/trace_collection/single-step-apm/ [3]: /tracing/trace_collection/single-step-apm/compatibility/ -[4]: /tracing/trace_collection/dd_libraries/nodejs/ [5]: https://app.datadoghq.com/fleet [6]: /tracing/troubleshooting/tracer_debug_logs/ [7]: /getting_started/tagging/unified_service_tagging/ [8]: /tracing/guide/injectors/ [9]: /tracing/trace_collection/single-step-apm/#instrument-sdks-across-applications -[10]: /tracing/trace_collection/single-step-apm/kubernetes?tab=agentv764recommended#remove-apm-for-all-services-on-the-infrastructure +[10]: /tracing/trace_collection/single-step-apm/kubernetes/#remove-instrumentation-for-all-services [11]: /tracing/trace_collection/single-step-apm/docker#remove-apm-for-all-services-on-the-infrastructure [12]: /tracing/trace_collection/single-step-apm/linux#remove-single-step-apm-instrumentation-from-your-agent [13]: /tracing/trace_collection/single-step-apm/windows#remove-single-step-apm-instrumentation-from-your-agent [14]: https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/#syntax-and-character-set [15]: /tracing/trace_collection/dd_libraries/dotnet-core/#installation-and-getting-started +[16]: #injection-appears-successful-but-traces-are-missing +[17]: #injector-debug-logs +[18]: /tracing/guide/injectors/#per-runtime-instrumentation +[19]: #troubleshoot-injection-in-datadog-fleet-automation +[20]: /tracing/trace_collection/single-step-apm/linux/#define-instrumentation-rules diff --git a/content/en/tracing/trace_collection/single-step-apm/windows.md b/content/en/tracing/trace_collection/single-step-apm/windows.md index e9760192d84..97e1a0062a0 100644 --- a/content/en/tracing/trace_collection/single-step-apm/windows.md +++ b/content/en/tracing/trace_collection/single-step-apm/windows.md @@ -1,5 +1,6 @@ --- title: Single Step APM Instrumentation on Windows +description: "Enable Datadog Single Step Instrumentation for .NET applications in IIS or Java and .NET applications across a Windows host." code_lang: windows type: multi-code-lang code_lang_weight: 30 @@ -18,7 +19,7 @@ With Single Step Instrumentation (SSI), you enable APM for your Windows applicat - Instrumenting .NET applications on IIS is generally available (Agent v7.67.1+, .NET SDK v3.19.0+). - Host-wide instrumentation of Java and .NET applications is in Preview. -
Before you begin, confirm that your environment is supported in the SSI compatibility guide.
+Before you begin, confirm that your environment meets the [SSI compatibility requirements][8]. ## Enable APM on Windows @@ -75,7 +76,7 @@ if ($p.ExitCode -ne 0) { By default, SSI installs the latest supported versions of the Datadog .NET and Java SDKs. To pin specific versions, add the `DD_APM_INSTRUMENTATION_LIBRARIES` property, for example `DD_APM_INSTRUMENTATION_LIBRARIES="dotnet:3,java:1"`. -Host-wide SSI instruments all Java applications on the host and all .NET applications running in IIS. To instrument .NET applications running outside of IIS, or for granular control over which processes are instrumented, [define instrumentation rules](#define-instrumentation-rules). +Host-wide SSI instruments all Java applications on the host and all .NET applications running in IIS. To instrument .NET applications running outside of IIS, or for granular control over which processes are instrumented, [define instrumentation rules][9]. {{< collapse-content title="Alternate: install with the in-app wizard" level="h4" >}} @@ -116,7 +117,7 @@ If you already have a Datadog Agent installed, use Fleet Automation to enable SS 1. Click {{< ui >}}Next{{< /ui >}}. 1. Review your configuration and click {{< ui >}}Deploy Configuration{{< /ui >}}. -For host-wide instrumentation, you can [define instrumentation rules](#define-instrumentation-rules) to instrument .NET applications running outside of IIS or to control which processes are instrumented. +For host-wide instrumentation, you can [define instrumentation rules][9] to instrument .NET applications running outside of IIS or to control which processes are instrumented. ## Verify your first trace @@ -150,12 +151,18 @@ To enable products, [set environment variables][3] in your application configura ## Remove Single Step APM instrumentation from your Agent -To disable SSI for .NET on your host, run: +### IIS-only instrumentation -```shell +To remove the SSI-managed .NET SDK package, run the following command in an elevated PowerShell session, then recycle the affected IIS application pools: + +```powershell &"C:\Program Files\Datadog\Datadog Agent\bin\datadog-installer.exe" remove datadog-apm-library-dotnet ``` +### Host-wide instrumentation (Preview) + +Removing only the .NET SDK package does not disable the Windows host injector or remove the Java SDK. To remove host-wide instrumentation, roll back the SSI configuration through Fleet Automation or follow the offboarding instructions provided with the Preview, then restart affected applications. Contact the Preview team or [Datadog Support][10] if the rollback control is not available in your account. + ## Advanced configuration {{< collapse-content title="Define instrumentation rules" level="h3" id="define-instrumentation-rules" >}} @@ -177,7 +184,7 @@ To configure instrumentation rules: 1. Define instrumentation rules: 1. Click {{< ui >}}Add New Rule{{< /ui >}}, then choose {{< ui >}}Allow Rule{{< /ui >}} or {{< ui >}}Block Rule{{< /ui >}} to specify whether matching processes should be instrumented. 1. Name your rule. - 1. Add one or more conditions. See [Define rule conditions](#define-rule-conditions) to learn more. + 1. Add one or more conditions. See [Define rule conditions][11] to learn more. {{< img src="tracing/trace_collection/define_instrumentation_rule.png" alt="The instrumentation rules UI, showing configuration options for defining a rule" style="width:100%;" >}} @@ -246,3 +253,7 @@ If you encounter problems enabling APM with SSI, see the [SSI troubleshooting gu [5]: https://app.datadoghq.com/apm/service-setup/workload-selection [6]: https://app.datadoghq.com/fleet/agent-management [7]: https://app.datadoghq.com/apm/services +[8]: /tracing/trace_collection/single-step-apm/compatibility/ +[9]: #define-instrumentation-rules +[10]: /help/ +[11]: #define-rule-conditions From deb3823fbff7f866ce0ca97e801d8a68fb1e3138 Mon Sep 17 00:00:00 2001 From: Brett Blue Date: Fri, 17 Jul 2026 16:02:55 -0600 Subject: [PATCH 12/13] Restore application instrumentation navigation --- config/_default/menus/main.en.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/config/_default/menus/main.en.yaml b/config/_default/menus/main.en.yaml index fa908a2f1c5..dc78b180dda 100644 --- a/config/_default/menus/main.en.yaml +++ b/config/_default/menus/main.en.yaml @@ -4284,7 +4284,7 @@ menu: parent: tracing identifier: tracing_glossary weight: 1 - - name: Set Up APM + - name: Application Instrumentation url: tracing/trace_collection/ parent: tracing identifier: tracing_trace_collection From a9d261095e2a1a2ca8ca5f2c40d67708447dec03 Mon Sep 17 00:00:00 2001 From: Brett Blue Date: Fri, 17 Jul 2026 16:10:15 -0600 Subject: [PATCH 13/13] Number compatibility reference links --- .../single-step-apm/compatibility.md | 23 +++++++++++-------- 1 file changed, 14 insertions(+), 9 deletions(-) diff --git a/content/en/tracing/trace_collection/single-step-apm/compatibility.md b/content/en/tracing/trace_collection/single-step-apm/compatibility.md index 28e852482fe..72b8e4e62e4 100644 --- a/content/en/tracing/trace_collection/single-step-apm/compatibility.md +++ b/content/en/tracing/trace_collection/single-step-apm/compatibility.md @@ -25,19 +25,21 @@ Select your environment to see compatibility requirements and limitations: ### Compatibility - **Status**: GA -- **Supported operating systems**: See [Linux distributions reference][linux-distributions] +- **Supported operating systems**: See [Linux distributions reference][1] - **Supported architectures**: x86_64, arm64 ### Requirements - Datadog Agent with APM Instrumentation enabled -- A [supported Linux distribution][linux-distributions] +- A [supported Linux distribution][1] ### Limitations - **SELinux**: Hardened SELinux environments are not supported. - **Small VM instances**: Very small instance types (for example, `t2.micro`) can experience timeouts. Use a larger instance type such as `t2.small` or higher. +[1]: #linux-distributions-reference + {{% /tab %}} {{% tab "Docker" %}} @@ -45,19 +47,21 @@ Select your environment to see compatibility requirements and limitations: ### Compatibility - **Status**: GA -- **Supported operating systems**: See [Linux distributions reference][linux-distributions] +- **Supported operating systems**: See [Linux distributions reference][1] - **Supported architectures**: x86_64, arm64 ### Requirements - Datadog Agent with APM Instrumentation enabled -- Docker running on a [supported Linux distribution][linux-distributions] +- Docker running on a [supported Linux distribution][1] ### Limitations - **Rootless Docker mode**: When running Docker in rootless mode, update the socket path in `/etc/datadog-agent/inject/docker_config.yaml` so SSI can connect to Docker. The default path is `/run/user/$UID/docker.sock`, but your environment may differ. - **Custom `runc` shims**: If your environment uses custom `runc` shims (for example, for GPU workloads), update the `runtimes` entry in `/etc/datadog-agent/inject/docker_config.yaml` to include both your custom runtime and the Datadog runtime required for SSI. +[1]: #linux-distributions-reference + {{% /tab %}} {{% tab "Kubernetes" %}} @@ -67,13 +71,13 @@ Select your environment to see compatibility requirements and limitations: ### Compatibility - **Status**: GA -- **Supported node pools**: Linux nodes only (see [Linux distributions reference][linux-distributions]) +- **Supported node pools**: Linux nodes only (see [Linux distributions reference][2]) - **Supported architectures**: x86_64, arm64 ### Requirements - [Datadog Admission Controller][1] enabled -- Kubernetes nodes running a [supported Linux distribution][linux-distributions] +- Kubernetes nodes running a [supported Linux distribution][2] ### Limitations @@ -81,6 +85,7 @@ Select your environment to see compatibility requirements and limitations: - **Windows pods**: For Kubernetes clusters with Windows pods, use namespace inclusion/exclusion or specify an annotation in the application to exclude them from library injection. [1]: /containers/cluster_agent/admission_controller/ +[2]: #linux-distributions-reference {{% /tab %}} @@ -121,7 +126,9 @@ Select your environment to see compatibility requirements and limitations: - Host-wide instrumentation is available only to Preview participants. - Language support is limited to Java and .NET. -For enrollment and setup instructions, see [Windows Single Step Instrumentation][windows-ssi]. +For enrollment and setup instructions, see [Windows Single Step Instrumentation][1]. + +[1]: /tracing/trace_collection/single-step-apm/windows/ {{% /tab %}} {{< /tabs >}} @@ -343,5 +350,3 @@ The following Linux distributions and architectures are supported for SSI across [1]: /containers/cluster_agent/admission_controller/ [2]: /tracing/guide/injectors/ -[windows-ssi]: /tracing/trace_collection/single-step-apm/windows/ -[linux-distributions]: #linux-distributions-reference